Biometric system

ABSTRACT

Some embodiments are directed to a biometric authentication system including headwear having a plurality of biosensors each configured to sample muscle activity so as to obtain a respective time-varying signal, a data store for storing a data set representing characteristic muscle activity for one or more users, and a processor configured to process the time-varying signals from the biosensors in dependence on the stored data set so as to determine a correspondence between a time-varying signal and characteristic muscle activity of one of the one or more users, and in dependence on the determined correspondence, authenticate the time-varying signals as being associated with that user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a national phase filing under 35 C.F.R. § 371 of andclaims priority to PCT Patent Application No. PCT/GB2018/050510, filedon Feb. 27, 2018, which claims the priority benefit under 35 U.S.C. §119 of British Patent Application Nos. 1703133.7 and 1711978.5, filed onFeb. 27, 2017 and Jul. 25, 2017 respectively, the contents of each ofwhich are hereby incorporated in their entireties by reference.

BACKGROUND

Some embodiments relate to a system for detecting facial biosignals forbiometric authentication. In particular, some embodiments relate tosystems for detecting facial muscle activity and facial skincharacteristics.

Several advanced technologies have been developed for biometricidentification, including fingerprint recognition, retina and irisrecognition, face recognition, and voice recognition. For example,Shockley et al., U.S. Pat. No. 5,534,855, generally describes usingbiometric data, such as fingerprints, to authorise computer access forindividuals. Scheidt et al., U.S. Pat. No. 6,490,680, describes identityauthentication using biometric data. Dulude et al., U.S. Pat. No.6,310,966, describes the use of fingerprints, hand geometry, iris andretina scans, and speech patterns as part of a biometric authenticationcertificate. Murakami et al., U.S. Pat. No. 6,483,929, generallydescribes “physiological and histological markers,” including infra-redradiation, for biometric authentication. However, these types oftechnologies have penetrated markets can be only in a limited fashiondue to complicated and unfriendly acquisition modalities, sensitivity toenvironmental parameters (such as lighting conditions and backgroundnoise), and high cost. In addition, due to complicated acquisitionprocedures, the foregoing technologies usually may require operatorattendance.

Fingerprint recognition is well-established and the most maturetechnology of the group. But it has several drawbacks: a fingerprintrecognition system cannot verify physical presence of the fingerprintowner and therefore is prone to deception, limiting its suitability foron-line applications; the optical sensor is a costly and fragile devicegenerally unsuitable for consumer markets; the system suffers fromnegative connotations related to criminology.

Retina scanning technologies are characterised by high performance.However, they may require high-precision optical sensors, and are notuser-friendly because they may require manipulation of head posture andoperate on a very sensitive organ—the human eye. The optical sensor isalso costly and fragile.

Iris and face recognition systems are user-friendly technologies sincethey record an image from afar and are not intrusive. However, they mayrequire digital photographic equipment and are sensitive to lightingconditions, pupil size variations and facial expressions. In addition,iris recognition performance is degraded by the use of dark glasses andcontact lenses, and face recognition may be deceived by impersonation.Face-based systems based on computer vision have been described when theimaging system is separate from the user and performs facial recognitionplus gesture recognition. An example of this is Wang et al (U.S. Pat.No. 9,202,105).

Voice recognition is the most user-friendly technology of the group.However, it may require a low-noise setting and is highly sensitive tointrinsically variable speech parameters, including intonation.Moreover, existing conventional recording technologies may be used todeceive speech-based recognition systems.

An ideal biometric should have very low intra-subject variability, veryhigh inter-subject variability, very high stability over time andreproducibility. It is conventionally known that biosignal-basedbiometrics can have unique features for each individual. For example,because everybody has different brain configurations, spontaneouselectroencephalograms (EEGs) between subjects are different. Thereforethere is a high inter-subject variability. A similar phenomenon has beenobserved for ECG as heart electrical characteristics, heart rate, rhythmand morphology vary between subjects. For example, Kyoso et al.(Proceedings of the 23rd Annual International Conference of the IEEE,Vol. 4, pp. 3721-3723, 2001) compare a patients electrocardiogram (ECG)with previously registered ECG feature parameters for purposes ofidentification. Despite the inter-individual ECG and EEG variabilitythat allows their use as a biometric identifier, their intra-individualvariability, which involves signal variations for the same subject,limits their robustness.

Thus, a need exists for a system for identity recognition that is atleast one of more reliable, robust, hard to deceive (on-line andoff-line), low cost and user-friendly. There is a need for an identityrecognition system that may be used in stand-alone applications orintegrated with existing security systems.

SUMMARY

Some embodiments provide a biometric authentication system including:headwear including a plurality of biosensors each configured to samplemuscle activity so as to obtain a respective time-varying signal; a datastore for storing a data set representing characteristic muscle activityfor one or more users; and a processor configured to: process thetime-varying signals from the biosensors in dependence on the storeddata set so as to determine a correspondence between the time-varyingsignals and the characteristic muscle activity of one of the one or moreusers, and in dependence on the determined correspondence, authenticatethe time-varying signals as being associated with that user.

The biosensors may be located on the headwear such that, in use, thebiosensors are configured to detect muscle activity of a predeterminedplurality of facial muscles. The predetermined plurality of facialmuscles may include a muscle that is involuntarily co-contracted in atleast one facial expression. The predetermined plurality of facialmuscles may include corresponding muscles from each side of a user'sface, and the processor may be configured to determine a measure ofasymmetry in the muscle activity of the corresponding muscles.

Suitably the data set includes a stored time-varying signal, and theprocessor is configured to process the time-varying signals independence on the stored data set by comparing the time-varying signalsto the stored time-varying signal. Suitably the data set includes one ormore stored signal features, and the processor is configured to processthe time-varying signals in dependence on the stored data set bycomparing one or more signal features of the time-varying signals to theone or more stored signal features.

Suitably the processor is configured to determine whether a thresholdnumber of signal features of the time-varying signals correspond to theone or more stored signal features. Suitably the one or more storedsignal features is determined in dependence on a previously-recordedtime-varying signal. Suitably the processor is configured to determinethat a signal feature of the time-varying signals corresponds to astored signal feature where the respective signal features are within apredetermined tolerance of one another. Suitably the processor isconfigured to determine that a signal feature of the time-varyingsignals corresponds to a stored signal feature where the signal featureexceeds a threshold value for that signal feature determined independence on the stored signal feature.

Suitably the biometric authentication system is configured to determinea first authentication level in dependence on determining that a firstthreshold number of signal features corresponds to the one or morestored signal features, and to determine a second authentication levelin dependence on determining that a second threshold number of signalfeatures corresponds to the one or more stored signal features, wherethe second threshold number is greater than the first threshold number,the second authentication level permitting authentication of the user toa higher security level. Suitably the biometric authentication system isconfigured to determine a first authentication level in dependence ondetermining that the respective signal features are within a firsttolerance of one another, and to determine a second authentication levelin dependence on determining that the respective signal features arewithin a second tolerance of one another, where the second tolerance issmaller than the first tolerance, the second authentication levelpermitting authentication of the user to a higher security level.

Suitably the characteristic muscle activity represented by the data setincludes a measure of one or more of: an amplitude of at least one ofthe time-varying signals; a variation with time of the amplitude of atleast one of the time-varying signals; a measure of asymmetry in theelectrical muscle activity of corresponding muscles from each side of auser's face; and the timing of activation of at least one facial muscle,relative to a common time base or to the timing of activation of atleast one other facial muscle.

Suitably the processor is configured to determine the correspondencebetween the time-varying signals and the characteristic muscle activityby at least one of: applying a pattern recognition algorithm; andapplying an analysis of variance test.

Suitably the processor is configured to process the time-varying signalsto obtain a plurality of representative values of the time-varyingsignals, the data set including a plurality of stored representativevalues, the representative values and the stored representative valuesbeing generated by a predefined algorithm, wherein the processor isconfigured to compare the representative values to the storedrepresentative values.

Suitably the processor is configured to compare the time-varying signalsto the stored time-varying signal by determining whether the signals arewithin a predetermined tolerance of one another.

Suitably the biosensors include one or more electric potential sensors.Suitably the electric potential sensors are dry contact electricpotential sensors. Suitably the headwear includes sensors fordetermining a user's gaze. Suitably the processor is configured toprocess the time-varying signal by at least one of: clipping the signal;signal denoising; applying a signal baseline correction; using onsetdetection; using data segmentation; applying a log-transform; andextracting one or more features from the signal. Suitably the processoris configured to use onset detection by applying thresholding to thesignal. Suitably the processor is configured to denoise the signal by atleast one of: signal filtering; signal conditioning; and signalsmoothing. Suitably the processor is configured to filter the signal byat least one of: applying a notch filter; and applying a band-passfilter.

Suitably the processor is configured to construct a model for patternrecognition. Suitably the processor is configured to train the model bya machine learning algorithm. Suitably the machine learning algorithmincludes a kernel machine. Suitably the kernel machine includes one ormore of a Support Vector Machine (SVM), a neural network, a discriminantanalysis method and a k nearest-neighbour (KNN) method.

The sampled muscle activity may be electrical muscle activity and thestored data set may include characteristic electrical muscle activityfor one or more users.

The plurality of biosensors may include one or more of EMG sensors andoptical sensors.

Suitably one or more of the plurality of biosensors are optical sensorsconfigured to capture a series of images of an area of skin of a user soas to obtain a time-varying signal characteristic of the area of skinimaged in the captured series of images; the data store is configuredfor storing a further data set representing skin characteristics for theone or more users; and the processor is configured to process the imagesignal in dependence on the stored further data set so as to determine acorrespondence between the image signal and the skin characteristics forthe one or more users, and in dependence on the determinedcorrespondence, to authenticate the image signal as being associatedwith one of the one or more users.

Suitably the system is configured to output an authentication signal forcontrolling access to a resource.

Some other embodiments provide a method for authenticating a user ofheadwear, the headwear including a plurality of biosensors configured tosample muscle activity, the method including: sampling, using each ofthe plurality of biosensors, muscle activity of the user to obtain arespective time-varying signal; processing the time-varying signals fromthe biosensors in dependence on a data set representing characteristicmuscle activity for one or more users so as to determine acorrespondence between the time-varying signals and the characteristicmuscle activity of one of the one or more users; and in dependence onthe determined correspondence, authenticating the time-varying signalsas being associated with that user.

Suitably the step of sampling muscle activity of the user includessampling muscle activity as the user performs one or more facialexpressions. Suitably the method includes generating a signal to cause auser to perform a predetermined sequence of one or more facialexpressions.

Some other embodiments provide a biometric authentication systemincluding: headwear including an optical sensor configured to capture aseries of images of an area of skin of a user so as to obtain an imagesignal characteristic of the area of skin imaged in the captured seriesof images; a data store for storing a data set representing skincharacteristics for one or more users; and a processor configured to:process the image signal in dependence on the stored data set so as todetermine a correspondence between the image signal and the skincharacteristics of one of the one or more users, and in dependence onthe determined correspondence, authenticate the image signal as beingassociated with that user.

Some other embodiments provide a method for authenticating a user ofheadwear, the headwear including an optical sensor configured to capturea series of images of an area of skin of a user, the method including:capturing, using the optical sensor, a series of images of an area ofskin of the user to obtain an image signal characteristic of the area ofskin imaged in the captured series of images; processing the imagesignal in dependence on a data set representing skin characteristics forone or more users so as to determine a correspondence between the imagesignal and the skin characteristics of one of the one or more users; andin dependence on the determined correspondence, authenticating the imagesignal as being associated with that user.

Suitably the step of capturing the series of images of an area of skinof the user includes capturing the series of images as the user performsone or more facial expressions. Suitably the method includes generatinga signal to cause a user to perform a predetermined sequence of one ormore facial expressions. Suitably the method includes processing thecaptured series of images so as to generate a composite image where thecomposite image images an area of skin larger than an area imaged in asingle image of the captured series of images.

Some other embodiments provide a biometric authentication systemincluding: headwear including a plurality of biosensors each configuredto sample a facial biosignal so as to obtain a respective time-varyingsignal; a data store for storing a data set representing facialbiosignal characteristics for one or more users; and a processorconfigured to: process the time-varying signals from the biosensors independence on the stored data set so as to determine a correspondencebetween the time-varying signals and the facial biosignalcharacteristics of one of the one or more users, and in dependence onthe determined correspondence, authenticate the time-varying signals asbeing associated with that user.

Suitably the time-varying signals include data representing at least oneof: electrical muscle activity; and a series of images of an area ofskin. Suitably the series of images of the area of skin includes one orboth of: a series of overlapping and/or abutting images permittinggeneration of a static image including a composite image formed from aplurality of images in the series of images; and a dynamic imageincluding information on how at least a portion of the imaged area ofskin changes through the series of images.

The headwear may further include one or more proximity sensors arrangedto, in use, measure distance between the headwear and the user's head,and the processor may be configured to process the time-varying signalsfrom the biosensors when the distance between the headwear and theuser's head as measured by the first proximity sensors is withinpredefined bounds indicating that the headwear is correctly positionedon the user's head.

The data set may further include information representing the predefinedbounds for at least some of the one or more users.

The headwear may be a pair of glasses and the one or more proximitysensors may include a proximity sensor at a nose bridge of the glasseswhich, in use, is arranged to measure a distance between the glasses andthe skin overlying the glabellar muscle of the user.

The headwear may be a pair of glasses and the one or more proximitysensors may include at least one proximity sensor at the distal end ofthe arms of the glasses which, in use, is arranged to measure a distancebetween the glasses and post-auricular skin overlying the ear.

Any feature of any aspect above can be combined with any one or morefeatures of any other aspect. Method features can be rewritten asapparatus features, and vice versa. These have not been written out infull here merely for the sake of brevity.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the presently disclosed subject matter will now bedescribed by way of example with reference to the accompanying drawings.In the drawings:

FIG. 1 is an illustration of headwear embodied in a pair of glasses;

FIG. 2A is an illustration of headwear embodied in a headset;

FIG. 2B is a sectional view along line A-A in FIG. 2A;

FIG. 3 illustrates possible sensor positions for facial biometricmeasurement;

FIG. 4A illustrates a remote device for use with the headwear;

FIG. 4B illustrates an interface between the headwear and anotherdevice;

FIG. 5 illustrates facial expressions;

FIGS. 6A to 6F illustrate 8-channel EMG signals of two different peopleperforming a left eye wink;

FIG. 7 illustrates proportional co-activation of the posterior auricularmuscle with certain expressions (smiling);

FIG. 8 illustrates an EMG-based facial expression recognition system forbiometric identification;

FIG. 9 illustrates a series of images captured by an optical sensor;

FIG. 10 illustrates an optical sensor; and

FIGS. 11A to 11D illustrate how facial EMG can be combined with otherparameters.

FIG. 12 shows a pair of glasses 1203 including one or more proximitysensor 1201 and one or more contact sensors 1202 arranged about the ear.

FIG. 13 illustrates detection of movement of the glasses relative to theskin at an optical flow sensor located at the temple of a pair ofglasses.

DETAILED DESCRIPTION

The following description is presented by way of example to enable aperson of ordinary skill in the art to make and use some embodiments.The presently disclosed subject matter is not limited to the embodimentsdescribed herein and various modifications to the disclosed embodimentswill be apparent to those with ordinary skill in the art.

The general principles defined herein may be applied to otherembodiments and applications without departing from the spirit and scopeof some embodiments. Thus, some embodiments are not intended to belimited to the embodiments shown, but is to be accorded the widest scopeconsistent with the principles and features disclosed herein.

Unless the context may otherwise require, throughout the specificationand claims which follow, the word “comprise” or “include” and variationsthereof, such as, “comprises” or “includes” and “comprising” or“including” are to be construed in an open, inclusive sense, that is as“including, but not limited to.” Unless otherwise defined, all or mostscientific and technical terms used below have the same meaning ascommonly understood by a person of ordinary skill in the art.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences unless the context clearly dictates otherwise. The meaning of“in” includes “in” and “on.”

The term “authentication signal” as used herein refers to the signalsent by the biometric device to an access point which may be a physicalaccess, an augmented reality access or access to a computer-basedenvironment. The authentication signal may be a binary encoded sequencetransmitted by wireless transmitter, such as a wireless protocol, orwired component, such as a cable or electrical wire.

As used herein, the terms set and subset refer to one or more items.

Human-computer interactions can be made more secure by the addition of alayer to augment existing password-based methods. Biosignal-basedbiometrics is a generation of physiology- and behavioural-based identityrecognition modalities with a very high level of security and isparticularly suited to wearable technology.

Biometric technology provides a reliable security system for identityverification and recognition of individuals based on their inherentcharacteristics in both modes of identification and verification orauthentication. The human features can include physiological orbehavioural traits. Physiological features include face, iris, andfingerprints whereas keystroke dynamics, gait and voice/speech arebehavioural features. However, these modalities are not robust enoughagainst falsification. Some instances of attacks in biometric securitysystems involve the use of latex for re-creation of a fingerprint, voiceimitation and the application of contact lenses copied from originaliris features. In other instances, authentication methods may beviolated by ‘over the shoulder’ snooping whereby a passcode, password orgesture is observed secretly. Therefore it is advantageous if the methodof authentication is discreet. In the case of virtual reality, where avirtual reality headset may be worn, the headset may obscure movementsof upper facial muscles such as occur when winking or blinking. Howeverthese gestures would be likely to be visible in a social virtual realitysituation, or may be read or inferred from the data transmitted fromheadset sensors. Therefore a method of ensuring that the authenticationcode is can be only known to the user would be a great advantage.

Face recognition systems may be spoofed by use of photographs or videosof the target face. Leviticus taught that a biometric system for accesscontrol may be triggered by movement (US 2015/0278499 A1), but thissystem may still be spoofed by a hacker wearing a mask. Yicong andcolleagues (US 20160071111) taught that a facial gesture may be used forauthentication by computer vision, and specified that more than onecamera should be used and an expression such as a wink could be usedwhich might make the system more secure. However, this method still maybe spoofed by the use of disguise and advanced make-up techniques.

Biosignal-based biometrics is a new generation of behavioural biometricidentity recognition modalities which has been introduced extensivelyduring the last decades. The biosignal of an individual is highlysecured and personal. Therefore, it is hard to mimic and counterfeit.Current biosignal-based biometrics mainly work using ECG and EEG. Forinstance, ECG signals may be used to evaluate or profile an individual'sphysiological and cognitive state. Directly or remotely acquired ECGsignals may be used to identify a subject.

A critical drawback of existing biometric identification systems is thesignal recording. To record an ECG with good quality, electrodes shouldbe placed on the subject chest which makes it difficult to use forpublic applications. Moreover, intra-individual (or intra-subject)variability, which involves signal variations for the same subject, isanother issue. ECG biometric readings taken in the presence of cardiacdisorders still present an issue with variabilities that are higher thandesired. An ECG-based identification system would need to be invariantto such physiological conditions, amongst others. ECG-based biometricsare sensitive to the subject's internal emotions (mood) as well asphysical conditions due to activation of the sympathetic nervous system.Further, the ECG signal, as it can be recorded from a conductive contactwithout the individual's knowledge, may be retransmitted to spoof asecurity system. EEG recording for accurate, reproducible results isalso difficult as it may require experts to put the electrodes overparticular positions on the scalp.

In addition to the mentioned problems, one of the important drawbacks inall or most previously proposed biosignal-based biometrics is the lowlevel of flexibility. It is not possible to change a subject's biometriccharacteristic or even to increase the number of the subject'scharacteristics. This could be a problem if we consider many systemswith different levels of security. The corruption of a system with a lowlevel of security could help an attacker to gain access to a system witha higher level of security. Besides, unlike passwords, a subject cannotchange their fingerprint, ECG, or EEG. These problems can be addressedby using a biometric with a high flexibility.

As discussed in examples below, an approach based on biosignals derivedfrom a subject's face would be a highly flexible method for biometricsecurity systems as it is difficult to spoof, unlike image-basedsystems. In examples below a system requiring a user to perform a facialexpression or sequence of facial expressions for personal identificationand/or authentication is described. In this technology, superficialmuscle activities produced by different facial expressions are recorded,for example through surface electrodes. Then, a series of biosignalprocessing techniques and machine learning algorithms can be implementedto analyse the data. The process reliably identifies and verifies theuser. This process is very robust against falsification.

The facial biosignal approach may be based on electrical characteristicsassociated with facial muscles, such as electrical muscle activity. Theelectrical muscle activity is suitably characteristic electrical muscleactivity associated with one or more facial expressions or facialgestures. Such electrical muscle activity can, for example, bedetermined from electromyography (EMG). The muscle activity may also bemeasured indirectly by measuring the activation of the muscle ofinterest.

Electromyography is an electrophysiological technique for evaluating andrecording physiological properties of muscles at rest and whilecontracting. EMG is performed using a device called an electromyograph,to produce a record called an electromyogram. An electromyograph detectsthe electrical potential generated by muscle cells when these cellscontract, and also when the cells are at rest. The electrical source isthe muscle membrane potential, which is about −70 mV. Depending on themethod used, the resulting measured potentials range typically betweenless than about 50 μV to about 30 mV. Amplitudes of EMG signalstypically range between 0 and 10 mV (peak-to-peak), or 0 and 1.5 mV(RMS). The frequency of the EMG signal is typically between 0 and 500Hz. The usable energy of EMG signals is typically dominant between 50and 150 Hz.

EMG is a good technique for distinguishing between different subjectsfor several reasons. The motor unit firing pattern, motor unitrecruitment order and characteristics of the muscles change from personto person, and therefore electromyograms (EMGs) can be used foridentifying a person. The characteristics of EMG signals are differenteven though the appearances of gestures might look identical between twosubjects (people). EMG measurements are influenced by one or more of themuscle fibre pattern, motor unit discharge pattern, differences in bloodflow, neural activity, and neurotransmitter activity within the muscle,skin conductivity, position, shape and size of the muscle, developmentof muscle, motor unit paths, differing bone densities, distribution ofheat in the muscle, skin-fat layer, and gesture style. These factorsmake biosignal-based identification and authentication difficult tospoof.

Unlike conventional biometric modalities, the bioelectrical signals arehighly confidential and personal to an individual and thereforedifficult to forge.

Systems for determining electrical muscle activity, such as EMG systems,can therefore be used to reduce the effects of at least some of thedrawbacks mentioned above in the context of conventional systems.

Facial EMG-based systems offer highly flexible biometric approaches forboth individual identification and security systems. The facialEMG-based biometric technology might be applied in government andcommercial environments, secure public and travel documents, healthmonitoring systems, forensic systems (including identifying criminalsand law enforcement) and distributed systems such as smart cards and/ormany others, including for example gaming systems for profile selection.A facial expression can lead to the generation of particular electricalmuscle activity (i.e. related to that facial expression, such ascharacteristic activity of muscles used in pulling the particular facialexpression), which can be used to identify and/or authorise anindividual. A sequence of facial expressions will provide even bettersecurity since this would lead to the generation of particularelectrical muscle activity of greater complexity. The greater the numberof expressions, the greater the complexity, and the greater thesecurity.

The present techniques are based on measuring biosignals from the face.As an exemplar we describe EMG, however other biosignals measurable fromthe face may be used including but not limited to ECG,electrooculography (EOG), EEG, optical image signals or a combination ofthese with or without other data from sensor modalities such asinformation from proximity sensors and/or contact sensors. For example,EMG and EOG can be used in combination with one another. Such acombination of techniques can provide greater flexibility and/orenhanced security.

There are several issues that must or should be considered wheninvestigating whether a feature is suitable for use as a biometric.These are discussed below in the context of electrical muscle activity,with EMG as an example of determining such electrical muscle activity.

-   -   Reproducibility—a feature should be highly reproducible for use        as a biometric. Facial expressions or gestures can be reproduced        by an individual due to the consistent muscle origins and        insertion points on the face. Further, the present techniques        utilise the fact that facial gestures and expressions exhibit        certain stereotyped behaviours. In particular, unilateral        expressions, such as an eye wink, are accompanied by involuntary        co-contractions of one or more additional muscle or muscle        group. For example, in the case of an eye wink, there is a        co-contraction of the opposite frontalis muscle to maintain        opening of the opposite eye. Force contractions tend to create        “overflow” effects whereby non-target muscles are recruited as        part of the target activation. For example forced eye closure        recruits the zygomaticus muscle of the cheek, which in turn        leads to co-contraction of the peri auricular muscles (FIG. 7).        These co-contractions are also reproducible. In terms of        security, the sensor outputs resulting from such unilateral        voluntary expressions are harder to replicate.    -   Stability over time—a feature should be highly stable over time        for use as a biometric. Slowly-varying features may be used,        where the slow time variation can be taken into account (for        example as part of a machine learning process). For example a        skin patch measured with an optical sensor may change slowly,        but remain consistent enough between uses on a daily basis such        that the system can be regularly updated. However, features that        vary quickly are less suitable since the higher variability        would be likely to mean that the same individual might not be        recognised as such.    -   Controllability—a feature should be highly controllable for use        as a biometric, since it is desirable that the feature can be        obtained as and when desired. Unlike EEG signals facial gestures        are at least partially under easy voluntary control, and so a        good candidate for use as a biometric.    -   Subject presence—a feature should be highly indicative of the        presence of a live subject, in person. Biosignal-based        biometrics have this benefit.    -   Universality—a feature or biometric characteristic should be        universally applicable. Biosignal-based biometrics have this        benefit. It is possible to obtain signals from people who are        alive.    -   Uniqueness—a feature should preferably or advantageously be        unique for use as a biometric. The biometric characteristic        should be able to unequivocally identify the individual(s)        within the target population. Regarding facial muscle activity,        the motor unit firing pattern, motor unit recruitment order        and/or characteristics of muscles such as size, location,        movement pattern, origin and insertion point) changes from        person to person, and therefore can suitably be used for        identifying a person.    -   Collectability—a feature or biometric characteristic should be        quantitatively measurable. Glasses based biometrics have this        benefit given their wide use.    -   Acceptability—Glasses are the most popular wearable    -   Flexibility—Unlike ECG- and EEG-based biometrics, facial EMG        provides a flexible biometric as it is possible to change the        biometric characteristic to set different levels of security,        for example by combining different facial expressions in one or        more sequence. Fingerprint data, ECG data, and EEG data cannot        be easily changed at will, but it is possible to produce        different facial expressions, alone or in combination with other        facial expressions, easily.

The above discussion shows that electrical muscle activity of facialmuscles is a suitable candidate for a biometric for distinguishingbetween subjects.

In order to reliably perform biometric identification based on facialmuscle activity of a subject, one or more biosensors, such as surfaceelectrodes or imaging sensors, may be positioned at a known location inthe vicinity of the muscles to be activated. This can be achieved bymounting the sensors to, or embedding the sensors in, a head-mounteddevice or headwear such as one or more of a helmet, mask, balaclava, oreyewear, for example in the style of goggles, glasses, sunglasses,protective eye shields and so on. Alternatively or additionally thesensors may be mounted in or on a headset, for example an audio headsetincluding earphones (such as for listening to music or making telephonecalls) an ear-mounted device such as headphones, earbuds or atelecommunication device. In this case the sensors can suitably beactivated by pericranial muscles. The latter includes, but is notlimited to, posterior auricular, superior auricular, anterior auricularmuscles, the temporalis muscle, masseter and occipital muscles.Suitably, “headwear”, includes any wearable device that in use at leastpartly overlies facial muscles of the user. FIGS. 1 and 2 show exampleconfigurations for a glasses-mounted system (FIG. 1) and a goggle (orvirtual reality)-mounted system (FIG. 2A). These will be described inmore detail below.

The use of a system based on headwear or facewear, and which may beincorporated into glasses, goggles or other facewear, is particularlyadvantageous for Virtual Reality (VR) and augmented reality uses.

Biosensors suitable for use as EMG electrodes may be standard wet (e.g.silver chloride gel) electrodes with adhesive pads, or dry electrodes.The latter may be resistive electrodes or capacitive electrodes which ineither case may present metallic, conductive fabric or conductivesilastic or rubber coated exterior surfaces for contact with a subject.The sensors can suitably be electric potential sensors (e.g. a PlesseyEPIC sensor) and/or surface EMG (sEMG) sensors (e.g. contact sensorsmanufactured by mc10 or Toumaz), and are suitably configured to detectelectrical activity of muscles. Preferably or advantageously the sensorsare electric potential sensors because these exhibit high sensitivityand may not require a conductive medium such as a gel or conductiveadhesive patch to electrically couple the sensor to the skin. Thesensors can include dry contact electric potential sensors. Muscleactivity may alternatively or additionally be detected from the motionof the skin overlying the muscle using a body-worn imaging sensor. Acombination of types of sensor can be used.

In the example illustrated in FIG. 1, the headwear is a pair of glasses100. The glasses 100 include a frame body 101 supporting a pair oflenses 102 in any suitable configuration. The lenses 102 could be madeof any suitable material, including glass and plastic (such aspolycarbonate). The lenses 102 may be prescription lenses. The glassesinclude two arms 103 extending from the frame body to the earpieces 104which are configured to hold the glasses in place at or around the earsof a user. The arms could form a single piece with the frame body, orthe arms could be foldable towards the frame body so as to allow theglasses to be stored more compactly. The glasses also include anosepiece or nosepieces 105 so as to allow the glasses to sitcomfortably at the nose of the user.

The glasses 100 include a set of biosensors for detecting the facialmuscle activity of a user. By arranging the biosensors at particularpoints on the glasses, the activity of different facial muscles can bedetected, such as a predetermined plurality of facial muscles. Thebiosensors, and additional components of the glasses 100, areschematically illustrated in FIG. 1. For example, biosensors 106, 108(such as electric potential sensors) could be provided at nosepiece(s)105 such that, when the glasses are worn, the biosensors engage with oneor more of the glabellar and/or frontalis muscles (preferably oradvantageously by a biosensor 108) and nasalis muscles (preferably oradvantageously by biosensors 106). In the example illustrated in FIG. 1,one biosensor 108 is provided for sensing the glabellar and/or frontalismuscles. In other examples, one or more additional such biosensor may beprovided. Additionally or alternatively, biosensors 110 (such as dryelectrode expression sensors) could be provided at the earpieces 104(i.e. at or towards the distal ends of the arms of the glasses) suchthat, when the glasses are worn, these biosensors engage with one ormore of the anterior auricular, superior auricular and posteriorauricular muscles. Additionally or alternatively, biosensors 112 couldbe provided at the frame body 101 to allow the activity of theorbicularis oculi muscles to be detected. Additionally or alternatively,biosensors can be provided on the supporting arms of the glasses andarranged, in use, for detecting the activity of the temporalis and/ormasseter muscles. Optical or other kinds of biosensors 114, arranged fordetecting electrical activity in, or changes in the topology of, thezygomaticus and/or levator labii muscle, can additionally oralternatively be provided. Optical biosensors will be described in moredetail below. There could be more than one biosensor and optionally morethan one type of biosensor at each of the exemplary locations shown inFIGS. 1 and 2A.

The glasses illustrated in FIG. 1 are portable headwear for detectingthe activity of one or more muscle or group of muscles. To power thebiosensors a battery 120 is provided. The battery 120 may be replaceableand/or rechargeable. For example, the battery 120 may be a 12 hourrechargeable battery. A port 122 is suitably provided for recharging thebattery. The glasses, as illustrated in the example shown in FIG. 1, areprovided with a data acquisition system 124 for acquiring data from thebiosensors. The glasses are suitably also provided with a dataprocessing system, though this may not be necessary in all or mostexamples. Suitably, where the data processing system is provided at theglasses, the data processing system is provided together with, or aspart of, the data acquisition system 124. Suitably, the glasses areprovided with a high quality data acquisition and processing system 124.The glasses may include a data store for storing a data set representingcharacteristic muscle activity for one or more users. Suitably, the datastore is provided at or adjacent the data acquisition system 124.Suitably, the battery 120 and data acquisition system 124 are providedin the arms of the glasses. This can aid the weight distribution withinthe glasses frame and make the glasses more comfortable to wear. Inother examples, the battery 120 and data acquisition system 124 can beprovided at any suitable location in the glasses frame 100.

The port 122 is suitably a combined power and data port, for example aUSB port such as a micro-USB port. Where the battery 120 is notrechargeable, the port 122 may be a data-only port. The port 122 thuspermits data to be transferred to and from the glasses 100. Datagenerated by the biosensors may thus be transferred from the glasses 100to another device, as will be discussed later. Data may also usefully betransferred to the glasses 100. For example, where the glasses 100include a data processing system, the data processing system willsuitably include logic and/or processing algorithms for processing data.This logic and/or these algorithms are suitably programmable and/orupdateable through a wired data connection made via the port 122.Additionally and/or alternatively to the data port 122, the glasses 100suitably include a wireless module 130 for connecting the glasses 100wirelessly to another device. Suitably the wireless module includes aBluetooth module, for example a Bluetooth module configured to operateaccording to the Bluetooth LE (low energy) wireless protocol. Suitablythe wireless module 130 includes a dual mode Bluetooth module. Othertypes of modules and/or wireless protocols, such as other Bluetoothprotocols and Wi-Fi, are possible. Bluetooth LE is particularly suitabledue to its low power requirements that may be necessary. Typically theother device with which the glasses 100 are to communicate wirelesslywill be well within the range of Bluetooth LE. This can also aid datasecurity by restricting the broadcast of the data to within the rangesupported by Bluetooth LE. Use of a low energy protocol can assist inadvantageously prolonging the life of the battery 120.

In another example, as illustrated in FIG. 2A, the headwear can beembodied in a headset such as a virtual reality headset 200. Biosensors,for example one or more biosensor as described in connection with FIG.1, can suitably be provided at the headset 200. For example a biosensoror a pair of biosensors 206 can be provided at a low front centralportion for engaging, in use, with the nasalis muscles. A biosensor 208can be provided at the front centre of the headset for engaging with oneor more of the glabellar and frontalis muscles. One or more biosensor212 can be provided above a region provided for a lens or screen forallowing the activity of the orbicularis oculi muscles to be detected.One or more biosensor 214 can be provided at a side portion of theheadset for detecting electrical activity in, or changes in the topologyof, the zygomaticus and/or levator labii muscle. It will be understoodthat these biosensors can be provided in any suitable combination in theglasses 100 and/or the headset 200.

The glasses 100 and/or the headset 200 may also include a heart ratemonitor 226 (as illustrated in FIG. 2A) and/or a motion sensor 228 (asillustrated in FIG. 1). The motion sensor 228 may include an inertialmeasurement unit. The motion sensor may include an accelerometer, suchas a 6-axis accelerometer. The motion sensor may include a gyroscope.Additional sensors and/or types of sensors may also be provided topermit the capture of data in addition to the electrical muscle activityof a facial muscle. Such additional data can be used to enhance the useof the data as biometric data, as will be further described below.

Referring again to the example illustrated in FIG. 2A, the headset 200includes a thin, flexible support 240. The support may be 3D printed forease of construction. The support 240 suitably supports a material layer242. The material layer is suitably a layer of foam 242 or other soft orpadded material. This construction permits the headset 200 to belightweight. The soft or padded material 242 aids comfort when theheadset is worn against the face of a user. The material layer 242suitably has a laser cut profile. The flexibility of the support 240aids in conforming the headset 200 to the user's face. The support 240need not be flexible in all or most examples.

FIG. 2B illustrates a section through the headset 200 along the line A-Ain FIG. 2A. The foam 242 or other padding is provided on the support240. The material layer 242 suitably includes insert portions 244, 246.The insert portions can be used to support the sensor 250. Suitably theinsert portions are of a different thickness to the bulk of the materiallayer 242, for example as illustrated in FIG. 2B. A biosensor 250 isprovided on a foam portion 246 which is of a reduced thickness comparedto the bulk of the foam 242 such that the sensor 250 is recessed fromthe front surface of the bulk of the foam 242. The sensor 250 is coupledto the data acquisition system (not shown in FIG. 2B) by any suitablecoupler, for example conductive wires 252. A layer 254 is provided overthe sensor 250 to act as a protective cover. The layer is suitably aconductive material such as a conductive rubber. Suitably the exposed(front) surface of the layer 254 (i.e. the surface which will come intocontact with the face of a user when the headset 200 is being used) isflush, or substantially flush, with the exposed (front) surface of thebulk of the foam 242. In some examples, the surface of the layer 254 maybe slightly proud of the surface of the bulk of the foam, i.e. it mayprotrude slightly from the surface of the bulk of the foam. This canassist in ensuring that the layer 254, and hence the sensor 250, makesgood contact with the face of the user when the headset 200 is in use.In some examples, the bulk of the foam 242 is more compressible than theportion of the headset 200 that houses the sensor 250, such that as theheadset 200 is pressed against the face of a user, the relativelygreater compression of the bulk of the foam assists in providing a goodcontact between the sensor 250 (via the layer 254) and the face of theuser.

Comparing the examples illustrated in FIGS. 1 and 2A, the glasses 100may have, in some examples, fewer sensors than the headset 200. This isbecause the glasses 100 are typically worn by a user with a greaterpositional repeatability (i.e. the glasses 100 will typically fit auser's head more closely than the headset 200 and so will better alignwith the same position on the user's head each time they are worn). Thiscan increase the accuracy with which the sensors are located adjacentone or more muscle group, and so reduce the overall number of sensorsneeded on the headwear to be able to obtain sufficient accurate data foruse in a biometric system.

In the example shown in FIG. 1, the arms 103 of the glasses 100 arepreferably or advantageously foldable and the headwear is suitablyconfigured to become enabled for use (e.g. power up) on the arms beingunfolded into positions in which the glasses can be worn by the user.This can be achieved through the use of switches at the hinges (notshown) between the frame body and arms, and can help to conserve batterypower at the device.

For headwear such as a face-worn device including a nosepiece such asthe glasses 100 and/or the headset 200, it can be advantageous toprovide a proximity sensor at the bridge of the nosepiece (of theglasses 100) or at a portion of a rim (of the headset 200) in order todetect when the headwear is removed by the user. On the proximity sensordetecting that the headwear has been removed (e.g. when the distance tothe skin measured by the proximity detector exceeds some predeterminedthreshold), at least some parts of the device may be configured to entera sleep state. Optionally one or more sensor and/or the data acquisitionsystem may enter a sleep state. Optionally the data processor may entera sleep state. Additionally or alternatively, the headwear is suitablyconfigured to switch between a low power mode and an operating mode independence on whether signals from a plurality of the biosensorsindicate that the headwear is being worn.

The headwear may therefore provide a convenient and accurate system fordetecting facial muscle activity such as electrical facial muscleactivity which can be provided as a low power, portable system that canbe used whilst the user is moving about.

The glasses shown in FIG. 1 and the headset shown in FIG. 2A are merelyschematic and it is to be understood that the shape, style, size and fitof the glasses/headset may vary so as to allow the biosensors to engagewith the facial muscles of a given user in the manner described herein.Preferably or advantageously the glasses would be fitted by a healthcareprofessional or optician so as to ensure optimal engagement of thebiosensors with the muscles of the user. Similarly, the headset ispreferably or advantageously tailored (for example, by being adjustedand/or manufactured) to suit the face of a user, for example by being 3Dprinted to match the facial contours of a user. This need not be thecase in all or most examples.

FIG. 3 illustrates locations on the face of a user 300 at which musclesand/or groups of muscles can be found. At area 302, the frontalis musclecan cause a brow elevation movement. This movement is typicallyassociated with surprise, fear and/or looking upwards. Data may besuitably obtained using an EMG sensor and a motion sensor such as aninertial measurement unit (IMU). Whether or not a user is lookingupwards can be determined in dependence on the output from the motionsensor. For example, if the motion sensor indicates that the headwearhas tilted upwards, this might indicate that the user is lookingupwards, rather than showing surprise or fear. The heart rate monitorcan also be used to distinguish between these cases; for example anelevated heart rate might indicate fear.

At area 304, the depressor supercilii, procerus, and/or corrugatorsupercilii muscles can cause a brow lowering movement associated withfrowning, anger, concentration, pain, and/or a response to brightlights. A light sensor can be provided on the headwear in some examples.Data may be suitably obtained using an EMG sensor and optionally a lightsensor. This can help to distinguish whether the muscle activity is inresponse to a bright light. Data may be obtained using a proximitysensor and/or a heart rate monitor. The heart rate monitor can be usedto help assess whether the movement is due to a particular emotion, suchas anger, since emotions can cause changes in the heart rate of a user.

At area 306, the orbicularis occuli muscle can cause a closing of theeye, blink and/or wink movements associated with blinking and/orwinking, smiling and/or a fatigued expression. This muscle typicallyco-contracts with other facial muscles when a genuine smile isexpressed. Data may be obtained using an EMG sensor.

At area 308, the zygomaticus major muscle can cause lifting of thecorner of the mouth. This movement is associated with smiling, pain, atight eye closure, winking and/or certain phonemes such as “ee”. Datacan be obtained using an EMG sensor and/or a proximity sensor.

In the above examples, a proximity sensor may be used to detect changesin the relative distance between the skin and the sensor. For example,when the skin wrinkles as part of a facial expression, a portion of theskin may move closer to (or further away from) the sensor.

The headwear 100, 200 is suitably configured to couple to a remotedevice 400. The coupling may be by one or both of a wired and a wirelessconnection. The remote device suitably includes one or more of a smartwatch, a smartphone, a PDA or other mobile electronic device andoptionally one that includes a display, a laptop computer, a desktopcomputer and a server. The remote device may include a network accesspoint for permitting onward communication with a further computingdevice. Referring to FIG. 4A, the remote device 400 suitably includes acommunication port 402 for permitting communication between the remotedevice 400 and the headwear 100, 200.

The port 402 is coupled to a processor 408. Also coupled to theprocessor are a memory 410, a data store 412 and optionally also adisplay 414.

Suitably the port 402 permits bidirectional communication. In someexamples, more than one port can be provided. The port 402 suitablyincludes one or both of a wired connection module 404 and a wirelessconnection module 406 (both are illustrated in FIG. 4, though bothmodules need not be present in all or most examples). The wiredconnection module 404 can, for example, include a USB port. The wirelessconnection module 406 can, for example, include a Bluetooth module suchas a module configured to communicate using the Bluetooth LE protocol.Preferably or advantageously, the headwear 100, 200 includes a Bluetoothmodule and the remote device 400 includes a Bluetooth module, permittingthe headwear and the remote device to communicate with one anotherwirelessly using the Bluetooth protocol.

The memory 410 suitably includes a volatile memory and can be used tostore processing algorithms and/or start-up routines for the remotedevice 400. In some examples the memory 410 includes a non-volatilememory in addition or alternatively to the volatile memory. It ispreferable or advantageous for the remote device 400 to include avolatile memory to permit the stored algorithms and/or start-up routinesto be modified. Such modification can be achieved through a connectionmade via the port 402.

The data store 412 is configured to store a data set that representscharacteristic electrical muscle activity for one or more user, so as toenable the one or more user to be identified and/or authenticated byreference to the stored data set. The data set is suitably caused to bestored at the data store 412 during an enrolment or configuration phase,as will be described below.

Thus, a biometric authentication system includes headwear 100, 200 andoptionally also the remote device 400. Where the remote device is notpresent, the data processing may be carried out at the data processingsystem at the headwear. Where the remote device is present, the dataprocessing may be carried out either at the data processing system atthe headwear, at the remote device processor 408, or at a combination ofthe two.

The headwear 100, 200 may be connected to the remote device, such as acomputer system, in order to provide information on facial muscleactivity, such as electrical muscle activity, to the computer system.This could be for a range of purposes, including: for use in a virtualreality (VR) system (e.g. animating an avatar with the expressions madeby the user of the device); to provide biometric security (e.g.performing a predefined series of facial expressions can be used toauthenticate a user due to the unique pattern of muscle activationsinvolved in creating such a series of expressions); for hands-freecontrol (e.g. a user can provide input to a computer system throughselective activation of their facial muscles).

Typically, the headwear would be coupled to a computer system by adriver interface 450 provided at the computer system. Dashed line 455 inFIG. 4B indicates a possible boundary between the headwear 100, 200 andthe computer system. An application programming interface (API) 460 maybe supported at the computer system so as to provide a convenientinterface to other applications running at the computer system (e.g.games, VR software, and/or software for performing biometricauthentication).

Data is captured or sensed by the biosensors at the headwear. Suitably,the biosensors may be configured to sample muscle activity to obtain atime-varying signal. Suitably the data includes the time-varying signal.The time-varying signal may be characteristic of the electrical muscleactivity of one or more muscle or group of muscles, for example a muscleor muscle group which is, in use, near at least one of the biosensors.The ‘raw’ (i.e. unprocessed) time-varying signal can be stored at a datastore at one or both of the headwear and the remote device. Suitably thedata store stores a data set which represents characteristic muscleactivity for a user (for example a user from a group of users). The dataset can be a set of data which has been previously recorded by thebiosensors, or a data set which includes data obtained in dependence onpreviously recorded data, such as average data of a plurality ofpreviously recorded data or data extracted from previously recordeddata. The data set might, in some examples, include a set of parametersrepresenting the characteristic muscle activity, for example outputsfrom a machine learning model.

The characteristic muscle activity represented by the data set mayinclude a measure of one or more of an amplitude of at least one of thetime-varying signals, a variation with time of the amplitude of at leastone of the time-varying signals, a measure of asymmetry in theelectrical muscle activity of corresponding muscles from each side of auser's face, and the timing of activation of at least one facial muscle,relative to a common time base or to the timing of activation of atleast one other facial muscle.

The time-varying signal can be processed by the processor at one or bothof the headwear and the remote device. Suitably the processedtime-varying signal is stored at one or both of the data store at theheadwear and the remote device.

Since the raw time-varying signal is likely to may require a greateramount of storage than the processed time-varying signal, preferably oradvantageously the processed time-varying signal is stored. The rawsignal need not be stored. Where the remote device is present, suitablythe processed time-varying signal is stored at the remote device datastore 412. This approach reduces the amount of storage capacity neededat the headwear 100, 200, and so can reduce the cost and/or weight ofthe headwear.

In some examples, the raw signal can be processed at the headwear. Thiscan be advantageous where the amount of data needed to represent theprocessed signal is less than the amount of data needed to represent theraw signal. This is because transferring the reduced amount of data overthe wired or wireless connection can reduce the time taken, which meansthat an identification and/or authentication result can be obtained witha reduced delay, and/or reduce the power consumption of the headwear intransmitting this data, which means that the battery life of theheadwear can be prolonged. This can be particularly advantageous wherethe signal processing can be performed at the headwear with a relativelylow or minimal delay, and/or at a relatively low or minimal power cost.

Suitably, the time-varying signal can be processed at the remote device400 if desired, whether or not it has been processed at the headwear.Thus, any computationally heavy processing (which is likely to mayrequire additional resources and/or power) can be performed at theremote device 400 rather than at the headwear 100, 200. This can helpoptimise or reduce the overall time taken for the processing of thesignal.

Suitably, the processor is configured to process the time-varying signalin dependence on the stored data set. This can permit determination of acorrespondence between the time-varying signal and the characteristicmuscle activity of the user. The processor can be configured toauthenticate the time-varying signal as being associated with that userin dependence on the determined correspondence.

The biosensors can be located on the headwear such that, in use, theyare configured to detect muscle activity of a predetermined plurality offacial muscles. The predetermined plurality of facial muscles caninclude a muscle that is involuntarily co-contracted in at least onefacial expression, which can for example permit authentication to beperformed in dependence on the electrical muscle activity characteristicof this involuntarily activated muscle. The predetermined plurality offacial muscles can include corresponding muscles from each side of auser's face, and the processor is suitably configured to determine ameasure of asymmetry in the electrical muscle activity of thecorresponding muscles. This can permit authentication to be performed independence on the asymmetry (or a measure of the asymmetry) between theactivation of these corresponding muscles.

The processing by the processor of the time-varying signal suitablyincludes comparing the signal to a previously-recorded signal or storedtime-varying signal. The signal and the stored signal can be compared inraw (i.e. unprocessed, or substantially unprocessed) form or in modified(or processed) form. This permits flexibility in the comparison of thesignals, for example the processing power and/or algorithms that may berequired. The comparison of the signal with the stored signal caninclude determining whether a number of features of the signalscorrespond. The data set can include one or more stored signal features.Signal features of the time-varying signal can be compared to the one ormore stored signal features. Suitably the processor is configured todetermine whether a threshold number of signal features of thetime-varying signals correspond to the one or more stored signalfeatures. The threshold number of signal features can be selected asdesired to provide a comparison of desired accuracy. The higher thethreshold number, the higher the accuracy of the comparison. Features ofthe signal can be determined to correspond to one another where thefeatures are within a predetermined tolerance (for example within about15%, 10% or 5%) of one another. Signal features of the time-varyingsignal can be determined to correspond to one or more stored signalfeature where a signal feature exceeds a threshold value for that signalfeature, the threshold value being determined in dependence on therespective value of the stored signal feature. The processing caninclude determining whether the signal or signal feature and the storedsignal or stored signal feature are within a predetermined tolerance(for example within about 15%, 10% or 5%) of one another.

A first authentication level can be determined in dependence ondetermining that a first threshold number of signal features correspondsto the one or more stored signal features, and a second authenticationlevel can be determined in dependence on determining that a secondthreshold number of signal features corresponds to the one or morestored signal features. Suitably the second threshold number is greaterthan the first threshold number. The second authentication level canpermit authentication of the user to a higher security level. A firstauthentication level can be determined in dependence on determining thatthe respective signal features (of the time-varying signal and of thestored time-varying signal) are within a first tolerance of one another,and a second authentication level can be determined in dependence ondetermining that the respective signal features are within a secondtolerance of one another. Suitably the second tolerance is smaller thanthe first tolerance. The second authentication level can permitauthentication of the user to a higher security level.

The processor may be configured to process the time-varying signals toobtain a plurality of representative values of the time-varying signals.The data set may include a plurality of stored representative values.The representative values and/or the stored representative values may begenerated by a predefined algorithm. The processor may be configured tocompare the representative values to the stored representative values.The system may authenticate a user in dependence on this comparison.

In the following discussion, it is to be understood that where‘processing’ occurs, this processing can be performed at one or both ofthe headwear and the remote device. Similarly, a data set can be storedat one or both of the headwear and the remote device. However,preferably or advantageously, where the processing is performed at theremote device, the data set is stored at the remote device rather thanat the headwear. This avoids the need for the data set to be transmittedback to the headwear for storage. However, in some examples, it may bedesirable for the processing during an enrolment phase to occur at theremote device, to save on processing power at the headwear, but for thesubsequent identification and/or authentication of a user to occur atthe headwear. In such examples, the data set which is the result ofprocessing performed at the remote device may therefore be transmittedto the headwear for storage at the headwear so that this data set cansubsequently be accessed at the headwear.

The principle of operation of the system will now be described.Biometric recognition involves comparing an acquired biometric signature(such as a newly acquired biometric signature, i.e. a ‘live’ signature)against signature templates in a registered or enrolled biometricsignature template database. Additionally or alternatively, the acquiredbiometric signature can suitably be processed in dependence on thetemplate database and/or a stored data set which representscharacteristic electrical muscle activity for one or more users. Thiscalls for two phases of system operation: enrolment and recognition.

Additional layers of information may be used for enrolment orrecognition in combination with the biosignal. Such additionalinformation may include IMU data, head position, line of gaze, userlocation, presence of object in real, virtual or augmented space, and/oradditional biometric inputs such as voice, eye scanning, limb gestureand/or heart biosignals.

Suitably, in the enrolment phase, neuromuscular activity is recordedduring controlled facial expressions, based on a system of connectedbiosensors that reads the biometric signals through the skin, forexample by skin contact. These biosensors can be located on headwear,for example a helmet, mask, balaclava, visor, mask, glasses, earphonesand/or headset. At least one biosensor may also collect other biometricsignals including one or more of heart rate and rhythm, heart ratevariability, skin conductance level, skin temperature and so on.

In the enrolment, or calibration, phase, a user is instructed to selecta calibration mode (although in some examples, a calibration mode mightbe entered automatically, as desired). In this mode the system records arepresentative sample of time-varying data from the user which, in itsraw or processed form, can later be used to identify and/or authenticatethat user. Suitably, the user is prompted to carry out a series offacial expressions and/or head gestures, which might be one or moreexpressions and/or one or more gestures. Alternatively, the user cancarry out any series of facial expressions and/or head gestures. Thesystem is suitably configured to determine which expression or gestureis being performed and to cause the recorded signals to be associatedwith that user with the determined expression or gesture.

Options for gestures include but are not limited to a forced eyeclosure, a snarl, a narrow smile, puckering of the lips, a foreheadwrinkle or an eyebrow raise, a blink, a wider smile, a frown, a wink ofthe left eye, a wink of the right eye, a left smile, a right smile, aclenched jaw and a neutral expression. These expressions are illustratedin FIG. 5, as expressions E1 to E14. A jaw clench is particularly usefulas this gesture can be performed with the face in a neutral posture foradded discretion during identification and/or authentication.

Further, head gestures such as a nod, tilting the head sideways, shakingthe head and so on, can also be used. This permits an increase in thecomplexity that may be required for authentication.

As the user performs the series of expressions data is captured from thebiosensors. Suitably, as a user performs the expressions, the muscleactivity of the user's facial muscles are detected by the biosensors,and time-varying signals representing this muscle activity are generatedby the biosensors. The time-varying signal (or signals) are output bythe biosensors for storing and/or processing.

Suitably, a method for calibrating a biometric authentication system, ora method of authenticating a user, includes storing at a data storeparameters defining a pattern in the signal from the biosensors which ischaracteristic of a facial expression of a user. The stored parameterscan be processed to generate calibration information for each of thebiosensors defining a correspondence between the signal from thatbiosensor and the activity of the respective facial muscle. The methodmay include indicating to the user one or more facial expression for theuser to perform. The method may include entering a calibration mode inadvance of storing the parameters.

Once the recording is complete, the time-varying signal representing themuscle activity data (e.g. an EMG or optical data from skin overlyingthe muscle), and, if required, additional data such as an inertialmeasurement unit (IMU) signal, is analysed or processed. The processingsuitably reduces the amount of data needed to be stored. For example,periods at the start and/or end of the recording which relate to‘background’ signals rather than those generated by the expression (e.g.where the user has not yet started the expression or has finished theexpression) can be clipped since these will not generally representuseful data. A suitable algorithm can be used for such clipping.Suitably the processing of the time-varying signal results in a set ofelectro-biometric signature templates (subject-specific discriminatingfeatures) unique for each individual which are suitably used forconstruction of a database.

Examples of time-varying signals recorded from two users are shown inFIG. 6. Each of the illustrated plots includes 8-channel EMG datarecorded during a particular facial expression. In each case the signalsindicative of the expression appear just after halfway through theplots. The early part of the plots may therefore be clipped as mentionedabove, to save data processing and/or storage requirements that may benecessary. FIGS. 6A-6C illustrate signals recorded from a first user;FIGS. 6D-6F illustrate signals recorded from a second user.

Referring to FIGS. 6A-6C, the plots show that signal patternsillustrating muscle activity of muscles (some of which are involuntarilyactivated muscles) demonstrate similitude where the same user performsthe same expression a number of times. EMG is used in this example butit should be understood that other measurers of muscle activity could beused instead. Pattern similitude between different recordings of thesame user performing the same expression can also be seen from FIGS.6D-6F. Comparing FIGS. 6A-6C with FIGS. 6D-6F, it can be seen that thereare recognisable differences in relative signal properties between thefirst user and the second user.

These differences are characteristic of the user, and can be used todistinguish between users. However, rather than using the raw signals,it is preferable or advantageous to process the raw signals, and to usethe processed signals to distinguish between users.

For example, referring to FIGS. 6A-6C, in the first (uppermost) channel(smileR), the signal amplitude increases above the background level atapproximately halfway through the plot, returns to the background levelat approximately three quarters of the way through the plot. Thebehaviour of the signal amplitude of the first channel is consistentacross FIGS. 6A-6C. Thus this behaviour, on the basis of these figures,has a high intra-subject repeatability, and therefore appears to be auseful feature. This feature, or behaviour, of the channel (or, in moregeneral terms, of EMG data) can be extracted and used as acharacteristic feature, or behaviour, of that user. It can becharacterised as the signal amplitude of the first channel exceeding athreshold level, where the threshold level is set to be at or just abovethe maximum values typically attained during the ‘background’ portionsof the plot. Optionally, to reduce the effect of noise spikes causingthe first channel signal amplitude to exceed the threshold, the featurecan be characterised as the signal amplitude of the first channelexceeding a threshold level for at least a threshold period of time,and/or or the average (such as a time average) of the signal amplitudeexceeding a threshold level, optionally also for at least a thresholdperiod of time. The threshold level and the threshold period of time canbe set based on measured or expected values. For example, the thresholdlevel can be set at 1.5 times, 2 times, 5 times, 10 times, and so on,the maximum of the background signal recorded during a set-up phase. Thethreshold period of time can be set at some fraction of the expectedduration of a facial expression. For example, where a facial expressionmay last for 0.25 seconds (such as a wink), the threshold period of timecan be set at 0.25 times, 0.5 times, 0.75 times, and so on, thatduration. The threshold level and the threshold period of time can beset in many other ways, as would be apparent.

Features can be extracted from other channels. For example, referringagain to FIGS. 6A-6C, the fourth channel (smileL) and fifth channel(eyeL) might similarly lead to a characteristic of the respective signalamplitudes (or their averages) exceeding respective threshold levels(optionally also for respective threshold periods of time). The signalamplitude of the seventh channel (corrL) appears to increase and thendecay over a period of time. The behaviour of this channel mighttherefore lead to a characteristic for this channel of the signalamplitude (or its average) exceeding a respective threshold level, withthe signal amplitude (or its average) then decaying over a thresholdperiod of time.

The sixth channel (browL) in figured 6A-6C appears to remain atsubstantially background levels throughout the plot. This can lead to acharacteristic of the signal amplitude (or its average) for this channelnot exceeding a respective threshold level.

An electro-biometric signature template can be formed from one or morecharacteristics or signal behaviour. For example, a comparison of FIGS.6A-6C with FIGS. 6D-6F shows that the fourth channel (smileL) can beused to distinguish between these two users for this particularexpression: the signal amplitude of the fourth channel in FIGS. 6A-6Cdistinctly rises above the background level, whereas the signalamplitude of the fourth channel in FIGS. 6D-6F does not. Thus a templateformed using the characteristic of the fourth channel will be successfulin distinguishing between these two users.

Building up more complex templates, by increasing the number ofcharacteristics on which the templates depend, and/or by increasing thecomplexity of one or more characteristics (for example by being morespecific about relative amplitude increases and/or timings and so on),will lead to a greater level of discrimination between a larger group ofusers. For example, a template can be formed based on the fourth andfifth channels (smileL and eyeL) rising above respective thresholdlevels, the seventh channel (corrL) rising above its respectivethreshold level then decaying, and the sixth channel (browL) not risingabove its threshold level. Such relatively more complex templates arelikely to be able to distinguish more accurately between users. Ofcourse, a much greater complexity is possible than described here inthis relatively simple example.

Rather than needing to store all or most of the data received orrecorded as part of the time-varying signal, it is possible to limit thestorage requirements that may be necessary to a much more limited amountof data, for example one or more channel identifiers and thresholdvalues (for amplitude levels and/or periods of time). More generally,the characteristics of the behaviour of time-varying signals, such astheir amplitude levels (or amplitude levels relative to background orthreshold levels) and/or the time duration of amplitude level changescan be stored and/or used to form a template for a particular user, forexample for a particular expression.

The signal features might describe or include values abstracted from themuscle activity of a user, for example from a time-varying signal. Forexample, a set of signal features might be derived by applying analgorithm to a time-varying signal and storing parameters for thatalgorithm and/or algorithm outputs as a set of signal features. Thealgorithm might be a machine learning algorithm or an algorithmicrepresentation of a model formed using machine learning. Thus a data setrepresenting characteristic muscle activity for a user might include thealgorithm, the parameters for the algorithm, and/or algorithm outputsfor that user.

In the recognition (or identification and/or authentication) phase, thesubject interacts with the system in a similar manner to that of theenrolment phase. However a shorter recording time on the order of a fewseconds is sufficient. In many cases the recording time can be muchshorter than this. In general, the recording time can be as short as oneexpression (or as short as the desired number of expressions). Forexample, where a wink might last for 0.25 seconds, the recording timecan be as short as 0.25 seconds and still be able to capture the wholeexpression. In some cases, the recording time can be less than the timeduration of one expression. For example, where an expression is 2seconds long, say a smile, the characteristic behaviour might be able tobe extracted within a 1 or 1.5 second period. Therefore an accuratetemplate can be constructed without needing to record the entirety ofthe expression.

In general, the recording time in the recognition phase can be as shortas the time duration of a subset of expressions or gestures used in theenrolment phase.

In a verification procedure, the system processes the time-varyingsignal in dependence on the stored data set so as to determine acorrespondence between the time-varying signal and the characteristicmuscle activity. The system suitably then authenticates the time-varyingsignal as being associated with one or more authorised users independence on the determined correspondence. In one example, the systemis configured to process the acquired signals, form a biometric subjectsignature and compare it with the subject's enrolled biometric signaturetemplate.

In a subject identification procedure, the biometric data for a subjectin the data collection may be compared with the biometric data in thedata collection for some or all or most of the subjects included in thedata collection. This comparison may yield information that can helpidentify a subject as someone who is already included in the datacollection.

The facial gestures used in the recognition phase may be used in anycombination, for example a jaw clench, a neutral expression and a smile.The facial gestures may be used in any order, or in a specified order.Further, the facial gestures may be used with or without head gestureswhich may be used in combination with one or more facial gesture foradded complexity. For example, a sequence of wink, nod, clench.

Where facial gestures can be used in any order, the system is suitablyconfigured to recognise each facial gesture and to process the resultingsignal from that gesture accordingly. In one example, the user may (withor without prompting) perform a series of a number (three, for example)of facial gestures. In some examples, the identity of a user can bedetermined if a given number of those facial expressions (up to thetotal number in the series; two in one example) result in acorrespondence between the user and a known user.

Where facial gestures need to be used in a particular order, the systemis suitably configured to prompt the user to perform the requiredgestures that may be necessary in the required order that may benecessary. Alternatively, the user may perform the required series ofgestures without prompt. Such an approach can enhance the security ofthe system since identification/authorisation would then be based bothon ‘what you know’ (the order of the expressions) and ‘what you are’(the characteristic muscle activity of the user) criteria.

Any suitable number of facial expressions can be used. The greater thenumber, the greater the complexity and the higher the security of thesystem. In practice, this is likely to be balanced by theuser-friendliness of the system: requiring too many expressions willtake a long time and potentially exhaust or irritate the user.Statistically, a more complex template (based on a larger number ofexpressions) will enable discrimination between a larger pool of users.For a game console logon authentication system, where the system mayneed to distinguish between, say, 10 users, it may be sufficient to usea single expression. For a banking authentication system, where thesystem is likely to need to distinguish between a significantly largeruser group, and to a greater degree of accuracy, a series of four orfive expressions may be required to provide the desired level ofsecurity.

In general there will exist for each user a natural asymmetry betweenthe signals representing muscle activity between the left and rightsides of the face. Suitably the processor is configured to distinguishthis asymmetry. The presence and/or quantification of this asymmetry canbe used as a feature during the authentication process. For example, thefeature can include a ratio of signal amplitudes from muscles, such ascorresponding muscles, on each side of the face. Suitably the featurecan be determined to be satisfied if the ratio measured in therecognition phase is within a certain tolerance of the enrolment ratio.

The tolerance can be less than about 15% difference, less than about 10%difference or preferably or advantageously less than about 5%difference.

The highest frequency components of electrical muscle activity data suchas surface EMG signals are typically around 400-500 Hz. Convention thusrecommends sampling rates of 800-1000 Hz, particularly when used with ananti-aliasing filter with a high frequency cut-off of 400-500 Hz.Accordingly, the sampling frequency may suitably be set to 1000 Hz forEMG acquisition. However the sampling frequency may vary under differentcircumstances, for example if multiplexing the sensors.

A discussion of the ways in which the time-varying signal may beprocessed now follows. Any one or more of these processing approachescan be applied in any combination, as appropriate for a particularsituation and requirements that may be necessary, for example whetherprocessing power is less of a restriction and a very high level ofsecurity is desired (such as might be true of a financial authorisationprocess) or whether the identification should be low-power and quick(such as might be true when identifying the user of a games console froma limited number of users). As an exemplar of biosignal processing wedescribe EMG data processing, however data captured from the face mayinclude EOG or other biosignals as mentioned previously.

Clipping: The time-varying signal can be clipped to cut out portions ofthe signal that are recorded before (and/or after) the expression isbeing made. Such portions will not contain relevant data. Hence storingand/or processing those portions may not be necessary. Clipping thetime-varying signal to portions of interest can therefore lead toprocessing and storage efficiency gains. A suitable algorithm can beused to determine the relevant portions of the signal to be clipped. Insome examples, onset detection (mentioned below) can be used todetermine the start of a portion of the signal that is of interest. Theend of a period of interest can be determined, for example, bymonitoring an expected duration of an expression from a detected startpoint, and/or by determining when another threshold is crossed. Forexample, referring to FIG. 6A, when the signal amplitude (or itsaverage) falls below a threshold indicating that the signal has returnedto background levels, the system can determine that the expression hasended. This ‘end’ threshold may be the same as the threshold used todetermine when the expression starts (i.e. a ‘start’ threshold), or itcould be different, such as below, this start threshold. The use ofdifferent thresholds can avoid the system flipping between start/enddetections where noise is present in the signal as the signal amplitudecrosses the threshold or respective thresholds. Suitably, the differencein thresholds is at least the same as the maximum amplitude of the noiseon the signal, or a multiple of this maximum amplitude, where themultiple is greater than one.

Notch filtering: A notch filter is typically a filter that stronglyattenuates the power over a narrow region of the spectrum. Here, a notchfilter at 50 Hz or 60 Hz, depending on the local conditions, may be usedto filter out power line noise with minimal disruption to the rest ofthe signal.

Band-pass filtering: Filtering of the EMG signals may be used tooptimise the signal to noise characteristics. For example, EMGs may beband-pass filtered using a 6th order butterworth filter within the rangeof 30-450 Hz to cover the most significant spectrum of facial EMGsignals. However other filters may be used as conventionally known.

Onset Detection: Analysis of the surface EMG may require thedetermination of rapid transient changes in the activation patternindicating transitions between different motor-related processes thatcontribute to the production of a particular motor act. Differenttechniques can be applied in this step. A standard approach for on andoff time estimation is to determine the times at which the envelope ofthe signal exceeds a threshold. The threshold may be given byThreshold=μ+kσ

where μ and σ are the mean and standard deviation of the envelope duringa period of inactivity, and k is a constant. In one example, μ may be8.13, σ may be 5.32 and k may be 1.

Data Segmentation: Feeding most or all signals directly to a classifieris not usually practical due to the enormous amount of data and somenon-informative or artefactual EMG data. Therefore, to improveefficiency, signals can be mapped into lower dimension vectors (featurevectors) to highlight one of the important properties of EMGs. The datacan be segmented into a sequence of time portions for use in featureextraction. Since neuromuscular signals can include or can consist oftransient and steady states, a transient state is capable of classifyingthe myoelectric features which are extracted 100 msec after onset ofmuscle activation with high accuracy. However, an issue with thisapproach occurs while switching from class to class (i.e. from oneexpression or gesture to another) in real time (such as in real-timecontrol). In this case the transient state of the EMG signal may nothave enough information, for example if the expression is changedpart-way through the expression, and the system may be unable to detectthe type of facial expression correctly. Thus, where there are multipledegrees of freedom in complex systems, such as will occur where aplurality of facial expressions and/or gestures are used, the use of thetransient state may result in reduced accuracy in some situations, suchas might occur when an expression is changed before the transient stateof the EMG signal has enough information. Steady state data with 128msec segment length is much more accurate than transient data for handgesture classification, and it is more reliable when a faster systemresponse is needed.

In a generic framework, relatively longer segments tend to result inrelatively higher computational load and a greater propensity forfailure in real-time operations. Relatively shorter segments tend tolead to bias and variance in feature estimation. In light of this, toaddress real-time restraints, it is suggested that segment length shouldbe equal to or less than 300 msec. Further the minimum interval betweentwo distinct contractions of motions is almost 200 msec which means thatsegments longer than 200 msec should contain sufficient information forclassification. Segments of between about 200 and 300 msec length, forexample 256 msec length, may suitably be used with facial EMG signals.According to the requirements that may be necessary and computationalconstraints, non-overlapping or overlapping time segments of anysuitable length may be used depending on the level of security andperformance that may be required.

Feature Extraction: Classifying large numbers of EMGs createssignificant computational demand, and it is helpful to map EMG data intoa lower dimension. The purpose of feature extraction is to highlight thesignificant properties of EMGs and make the raw signals and/or processedsignals derived from the raw signals more meaningful for classification.There are numerous methods previously described with varying levels ofcomplexity and efficiency in diverse domains (time, frequency,time-scale) which represent different EMG signal characteristics. Forthe purpose of myoelectric classification, time-domain features aregenerally suitable due to simple representations of different motion andgesture characteristics and low computational load in real-timeapplications. In the present techniques, the Mean Absolute Value (MAV)of each segment may be used. The MAV can be computed as a feature by thefollowing equation:

${MAV}_{k} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}{x_{i}}}}$

where N is the length of the segment, k is the current segment, x_(i) isthe current point of signal and i is the index of the current point.

Pattern Recognition: Extracted features need to be classified intodistinct classes for the recognition of facial expressions. A variety offactors, such as electrode position, muscle fatigue, and skin surfaceconditions, cause variations in the EMG data acquired over time and mayresult in misclassification. Hence, a classifier should be able to copewith these issues. Moreover, the classifier must or should be fastand/or proficient enough to meet real-time restraints and classify thenovel patterns during the online training. A significant issue is thepotential variability in repeatably performing a specific voluntaryexpression.

An example of a pattern matching method is the Support Vector Machine(SVM) which is a kernel-based approach with a strong theoreticalbackground, and is a conventional tool for machine learning tasksinvolving classification and regression. A SVM constructs an optimalseparating hyper-plane in a high-dimension feature space of trainingdata that are mapped using a nonlinear kernel function. Therefore,although it uses a linear machine learning method with respect to thenonlinear kernel function, it is in effect a nonlinear classifier. Theuse of a nonlinear kernel function greatly increases the power oflearning and generalisation. Training an SVM involves the optimisationof a convex cost function; there are relatively few free parameters toadjust, and the architecture does not have to be found viaexperimentation.

SVM employs a regularisation parameter that allows adaptation tooutliers and error on the training set. Here, a multi-class SVM with aone-versus-one scheme is used along with a polynomial kernel function,and the penalty factor is, in some examples, tuned at 1000. Suitably, inthe present techniques, the muscles activated as part of the voluntaryexpression are used for pattern matching. As the spatial relationshipbetween the electrodes is fixed, the activation of other electrodesfollows a stereotyped pattern. An additional layer of security isachieved by performing pattern matching specifically focusing on theinvoluntary and antagonistic muscles that also activate. As an exemplar,during an eye wink, which is a voluntary contraction of one orbicularisoculi, the consensual activation of the opposite orbicularis oculi isantagonised by the contraction of the opposite frontalis muscle whichacts to elevate the brow and helps to keep one eye open. Therelationship between the voluntarily activated orbicularis, theinvoluntarily activated opposite orbicularis and the involuntarilyactivated frontalis are difficult to replicate, and form a dynamicbiometric. An example is shown in FIG. 6. Other examples of combinationsof active and involuntary muscle activations exist such asco-contraction of the peri auricular muscles with brow elevation orsmiling (as shown in FIG. 7).

Statistical Analysis: A reliable biometric trait should have lowintra-subject variability, high inter-subject variability, and highstability over time. In order to investigate inter-subject variability,a statistical significance test may be applied on one trial of aparticular facial expression of a plurality of subjects. An example thatcan be applied to this type of data is an analysis of variance (ANOVA).Such a test reveals if the facial EMG raw data or the extracted temporalfeatures are significantly different within the subjects (as shown inFIG. 7). Besides, by testing different types of facial expressions, itis possible to find the best or better facial expression thatstatistically can deliver the most significant difference withinsubjects. It is expected that such an expression is one which involvesco-contraction of antagonistic muscles. Another test can be designed toexamine the intra-subject variability by considering most or all trials(repetitions) of a similar facial expression performed by a subject.Very low variability for intra-subject trials indicates that facial EMGgenerated from facial expressions are reproducible.

Suitably, the processor is configured to perform signal pre-processingon at least one of the obtained time-varying signals (such as EMG data)including one or more of: signal baseline correction, signal denoisingand signal onset detection. Signal baseline correction can includeresetting the time-varying signals to a zero DC offset. Signal denoisingcan include increasing the signal to noise ratio by one or more ofsignal conditioning, signal filtering and signal smoothing. Signal onsetdetection can include applying thresholding to the signal. Thethresholds are suitably predetermined, and may vary between users. Forexample, during a machine learning process, the algorithms can determinea suitable threshold across most or all users, across a subset of users,or for an individual user. Suitably the threshold is approximately 10%,20%, 30%, 40%, or 50% of the peak value of the signal. Applying suchthresholding can mean that the system may only need process informativeportions of the time-varying signals, e.g. where the threshold has beenmet or exceeded.

A log transform may be applied to a time-varying signal in order to makethe feature vectors more separable. This can aid in feature extraction.Suitably the processor is configured to construct a model for each useror individual using the time-varying signal obtained in the enrolmentphase, and/or the features extracted from that time-varying signal.Suitably the processor is configured to evaluate the model for eachindividual using the time-varying signal obtained in the recognitionphase, and/or the features extracted from that time-varying signal.Suitably the model is constructed through an individual's facialenrolment or calibration signal or extracted features of that signal totrain a machine learning algorithm. The machine learning algorithm caninclude a kernel machine such as a Support Vector Machine (SVM), aneural network such as a radial basis function (RBF), a discriminantanalysis method such as linear discriminant analysis (LDA) and/or knearest-neighbour (KNN) for user identification and/or authentication.

In some examples, the system is configured so that the construction ofthe model for each individual incorporates synchronisation between anaction occurring in the real world (for augmented reality use) and/or avirtual world.

Once the time-varying signal (or signals) has been processed, theprocessed time-varying signal is suitably stored at a data store,accessible to at least one of a processor at headwear 100, 200 and aprocessor at a remote device 400. As mentioned above, the raw signalneed not be stored, though it could be. The stored time-varying signal,e.g. the processed signal, suitably forms at least part of a data set.The data set suitably represents characteristic electrical muscleactivity for one or more users, for example as determined during theenrolment phase.

The stored data set is available to the processor (at either or both ofthe headwear 100, 200 and the remote device 400). The dataset may be inany suitable form: e.g. as a set of relationships stored at a table orother data structure; as relationships defined in equation(s) oralgorithm(s) performed at the processor; and/or a machine learning model(and/or its parameters) used by the processor to perform muscle activitydetection and/or characterisation.

In examples of the present techniques, in the recognition phase, theuser is prompted to perform a predetermined series of facial expressions(which suitably includes more than one facial expression). The processoris configured to process time-varying signals obtained from thebiosensors so as to determine a correspondence between the time-varyingsignals and the characteristic electrical muscle activity. This ispreferably or advantageously achieved by processed the time-varyingsignals in dependence on the stored data set. In some examples, this canbe achieved by comparing the time-varying signal with a stored (orpreviously-recorded) time-varying signal. In some examples, this can beachieved by determining whether the time-varying signal matches a storedtime-varying signal. A match can be determined where the two signals arewithin a predetermined tolerance, say within 10% or preferably oradvantageously within 5% or more preferably or advantageously within 2%of one another.

Suitably, as discussed herein, processing the time-varying signal canresult in features, such as signal features, being extracted from thetime-varying signal. Thus, one or more extracted feature of thetime-varying signal can be used to determine whether to authenticate theuser. For example, the one or more extracted feature (or signal feature)can be compared with one or more stored extracted feature (or signalfeature). It can be determined whether the extracted features match orsatisfy the stored extracted features. A match can be determined wherethe extracted feature or features and the stored extracted feature orfeatures are within a predetermined tolerance, say within 10% orpreferably or advantageously within 5% or more preferably oradvantageously within 2% of one another. Satisfaction of the storedextracted features can be determined when the extracted features cross athreshold level (optionally for at least a predetermined period of time)with respect to the stored extracted features.

Where the system determines that the signals or extracted features aresufficiently similar to one another (noting that this level ofsimilarity can be selected in dependence on the processing, time and/orsecurity requirements that may be necessary of the authorisationprocess, with a greater similarity being that may be required for higherlevel security) where similarity can be determined based on the numberor proportion of extracted feature matches or extracted features thatsatisfy the respective stored extracted features, it can determine thatthe user is identified and/or authenticated. The system can generate anauthentication signal in response to such a determination. Theauthentication signal can provide the user access to functionality atthe headwear and/or at a remote computing device.

Where the system determines that the signals or extracted features arenot sufficiently similar to one another it can determine that the useridentification and/or authentication fails. The system can generate anon-authentication signal in response to such a determination, althoughit need not. The non-authentication signal (or a lack of anauthentication signal) can restrict user access to functionality at theheadwear and/or at a remote computing device.

FIG. 8 illustrates steps that might occur during an enrolment 802 andauthentication 852 process. During enrolment, a user will typicallyperform a facial expression 804 (or more than one different facialexpression). This can be as a result of a prompt from the system, or theuser can perform the facial expression without a prompt. Suitably, thesystem prompts the user regarding which facial expression to perform.The facial expression is preferably or advantageously performed morethan once by the user, so that the system can determine the variabilityand so on between different instances of the same expression from thesame user. In general, the expression is performed n times. The systemsuitably prompts the user to perform the expression n times. Suitablythe system prompts the user before each of the n times the expressionshould be performed. This can aid the system in determining the startand stop of each expression in the sequence of expressions.

During each of the expressions, the system records time-varying datafrom the user. In the example illustrated in FIG. 8, the system recordsfacial EMG measurements. Subsequent to recording the time-varying data,the system processes the recorded data. This is shown at 808 in theillustrated example, relating to EMG data processing. This processingcan include one or more processing steps. In the illustrated example,the processing includes applying a baseline correction 810, filtering812, onset detection 814, feature extraction 816 and featuretransformation 818. In other examples one or more of these processingsteps could be omitted or replaced with one or more other processingsteps.

Following the data processing, the system can use the processed data togenerate a reference template (or a series of reference templates) 820.The reference template, together with optional user data such as auser's name and so on, which can be input at 822, can be stored at adatabase 824 (such as at a data store as described herein). The data canbe used to construct a model for pattern recognition 826 for later useby the system in identifying or authenticating a user. The model can, insome examples, be constructed before the data is stored at the database.The model can be stored at the database. In some examples, the data neednot be stored, where the model is stored. The data and/or model storedat the database can be considered to be a data set that represents theuser and the facial expression or expressions performed.

The data set can suitably be used in a later authentication process 852.In the authentication process, a user performs one or more facialexpressions, for example in response to a system prompt or prompts. Thesystem can generate one or more signal to prompt the user. The one ormore prompt signal can generate an audio or visual prompt that directsthe user as to which expression to perform. The system recordstime-varying data from the user. In the example illustrated in FIG. 8,the system records facial EMG measurements 854. Subsequent to recordingthe time-varying data, the system processes the recorded data. This isshown at 856 in the illustrated example, relating to EMG dataprocessing. The processing steps can be as described above in thecontext of enrolment data processing 808. The processing steps can bethe same as or different from those in the enrolment phase. Suitably,the processing steps are the same in the enrolment phase and theauthentication phase. This can simplify the authentication process.

Following the data processing, the processed authentication data isinput to a pattern matching and model evaluation process 858 at whichthe authentication data can be processed in dependence on the storeddata set so as to authenticate the detected time-varying signal as beingassociated with one or more user who has been enrolled in the system. Anauthentication result can then be output by the system 860. Thisauthentication result can be used to control access to a resource.

The biosensors may, in some examples, include optical sensors. These canbe considered to be biosensors since they are able to detect signalscharacteristic of biological criteria, such as images of the skin and/orinformation describing the movement of skin as discussed in the examplesbelow. Such optical-sensing biosensors may be used additionally oralternatively to the biosensors described above. Optical sensors may beoptical flow sensors configured to output as a time-varying signal avector describing the direction and/or magnitude of skin movement oversome predefined time period which is representative of muscle activity.For example, an optical flow sensor may be arranged at wearableapparatus to measure the movement of an area of skin overlying a musclewhere the movement of that area of skin is representative of theactivity of that muscle. An example of an optical flow sensor is theADNS-9800 optical flow sensor which illuminates a surface whose movementis to be detected with infra-red light and monitors the resultingmovement using an image sensor (e.g. a charge coupled device or CMOSsensor). Movement may be detected by the image sensor through theapplication of a suitable optical flow algorithm (e.g. an algorithm fortracking the movement of contrast features within a sequence of framescaptured by the image sensor).

Facial features can include characteristic skin patterns or skin texturemaps: similarly to how a fingerprint is a particular pattern of ridgesand depressions on the skin of a fingertip, the skin's surface anatomyand micro-relief is unique. It is therefore a suitable candidate for usein distinguishing a user from another user and/or a group of usersand/or the general population. The skin's surface anatomy andmicro-relief is affected by both genetic factors (e.g. pigmentation,hair follicles, sebaceous pores, congenital lesions), environmentalfactors (e.g. sun spots, sun tanning) and localised factors (e.g. scars,infections and inflammation). Fine furrows intersect with each other,forming many types of shapes including especially triangles, andrhomboids, but also squares, rectangles, and trapezoids that contributeto the skin's glyphic patterns. The latter provide the skin with theflexibility required that may be necessary to allow various ranges ofmotion, particularly in mobile areas such as around joint regions. Skinwrinkles are different from glyphic patterns; wrinkles are largelygenetically determined and develop over time as a result of underlyingmuscular activity and dermal matrix deterioration. There are alsopassive skin wrinkles that occur due to repeated creasing of the skin,such as occur whilst sleeping (sleep lines).

Facial features can include one or more of the location, size,orientation, colour and so on of pigmented areas of the skin, hairfollicles, pores, lesions, scars and so on.

Facial features can include facial wrinkles, for example transient ordynamic wrinkles and/or static or permanent wrinkles. Facial wrinklescan develop for a variety of reasons and are generally classified aseither static or dynamic. Static wrinkles are the wrinkles and folds ofthe skin that are present when the face is at rest and the facialmuscles are not acting on the skin so as to form a facial expression.Dynamic wrinkles are those that occur due to the effects of facialmuscle activity on the overlying skin. With age, stress andenvironmental damage (such as sun exposure and smoking), dynamicwrinkles eventually become static wrinkles.

Due to the combination of skin surface anatomy, glyphic patterns andwrinkles, the skin represents a non-uniform surface microstructure withvaried reflectance.

Optical sensors may therefore be used to characterise facial skin, forexample by recognising one or more particular feature of the facialskin. The headwear 100, 200 may include at least one optical sensor aswell as or instead of other types of sensor. The optical sensors may beconfigured, in use, to capture an image of an area of the skin adjacentthe optical sensor. The optical sensors are suitably configured tocapture a series of images of an area of skin of a user so as to obtainan image signal which is characteristic of the area of skin imaged inthe captured series of images. The area of the skin can be imaged duringan enrolment phase, similar to the discussion above in the context oftime-varying electrical muscle activity signals. Thus, the system issuitably configured to store a data set representing skincharacteristics for one or more user. The data set can be stored at oneor both of the headwear 100, 200 and the remote device 400, as discussedherein. Suitably, where an optical sensor is provided, the processor isconfigured to process the image signal in dependence on the stored dataset so as to determine a correspondence between the image signal and theskin characteristics for the one or more users, and in dependence on thedetermined correspondence, to authenticate the image signal as beingassociated with one of the one or more users.

Suitably, the image signal is processed to extract at least a velocityof movement of the area of skin being imaged. In other words, adirector, or direction of movement, and a speed of movement can bedetermined in dependence on the image signal. Suitably the headwearincludes a processor configured to determine the velocity of movement ofthe area of the skin. The sensor may include the processor.

Such skin velocity can be used as a characteristic for distinguishingbetween subjects. Taking a frown as an example of an expression, ingeneral the movement of the brow in a frown is downwards. In somesubjects, the brow movement is more transverse and downward whereas insome other subjects, the brow movement is more vertical and downward.Determining the transverse and/or vertical component of the browvelocity and/or movement can permit distinguishing between such groupsof subjects.

Optical sensors are capable of visually sampling a surface (in this caseskin). The optical sensors are suitably capable of comparing samplescaptured apart in time so as to detect dynamic information, such asmovement of the surface relative to the sensor between those points intime. A visual sample captured by an optical sensor may be any kind ofrepresentation of the surface (e.g. a photograph) or representation oflight modified by the surface (e.g. an interference pattern or specklepattern created by the surface interacting with a laser directed at thesurface). Suitable algorithms may be used to track changes betweenvisual samples which represent movement of the sampled surface.

An optical sensor 1000 is shown in FIG. 10. The optical sensor includesa light source 1020, a light sensor 1022, and a processor 1024 forprocessing the output of the light sensor so as to determine atime-varying signal representative of dynamic information such as therelative movement between the sensor 1000 and the skin 1004. The lightsource 1020 is arranged to illuminate the patch of skin 1004 whosedynamic behaviour is to be detected. The light source may be, forexample, an LED or a laser. A suitable lens 1001 may be provided in thelight path so as to, for a given distance 1005 from the sensor to theskin, focus the light onto a patch of skin of suitable size which is tobe imaged. Similarly, a suitable lens 1002 may be provided in thereflected light path so as to, for a given distance 1005 from the sensorto the skin, focus the reflected light onto the light sensor 1022. Theprocessor 1024 may be configured to capture skin movement transverse, orsubstantially transverse, to the imaging direction of the optical sensor1000 using suitable algorithms configured to track movement of asurface. Algorithms of the type used in optical computer mice to trackthe relative movement of the mouse over a surface can be suitable.

Preferably or advantageously lens 1001 is a collimating lens so as toavoid the size of the illuminated area and the effective intensity ofthe light incident at the skin varying significantly as the distancebetween the skin and optical sensor varies. Lens 1002 is preferably oradvantageously an imaging lens selected so as to focus the reflectedlight from the patch of illuminated skin at the light sensor 1022 andallow the light sensor to capture the texture of the skin. One or bothof the lenses 1001 and 1002 may be replaced or augmented by adiffraction grating. The processor 1024 may be configured to performsuitable computational algorithms optimized to detect dynamic behaviourof the skin 1004 given the pattern created at the image sensor by thediffraction grating(s). A diffraction grating may be integrated with theimage sensor 1022 in order to provide a compact imaging unit. Forexample, the image sensor 1022 may be a Rambus Lensless Smart Sensor.

The light sensor may be, for example, a Charge Coupled Device (CCD) or aComplementary Metal-Oxide-Semiconductor (CMOS) sensor.

It can be advantageous to provide one or both of filters 1008 and 1009in the light path from the light source to the skin, and from the skinto the light sensor, respectively. Each of the filters 1008, 1009 couldbe any suitable kind of filter, such as a polarising filter and/or acolour filter. For example, the filters 1008 and 1009 could both bepolarising filters arranged to apply the same polarisation to the lightpassing through the filter. This arrangement can help the sensor todistinguish light which originated from the light source and wasreflected from the skin from ambient light. In a second example, filter1009 could be a polarizing filter oriented so as to pass plane-polarisedlight reflected from skin 1004 and attenuate light having otherpolarisations. Typically reflected light is at least partiallyplane-polarized with a direction of polarization parallel to thereflecting surface; the corresponding direction of polarization can beselected for a plane polarizing filter 1009. This can help to isolatereflected light from ambient light. In some examples, filter 1008 mayadditionally be provided as either a polarizing filter whose axis ofpolarization is substantially oriented at 90 degrees to filter 1009(i.e. cross-polarised with respect to filter 1009) or plane-polarizedwith a direction of polarization substantially parallel to thereflecting surface.

The optional filters 1008, 1009 may be coloured filters arranged to passfrequencies of light which are particularly advantageous for imagingskin and/or rejecting ambient light. Filter 1008 may be employed toprovide substantially monochromatic illumination if light source 1020emits a broad spectrum of colours. In the event that light source 1020is monochromatic (whether alone or when combined with filter 1008),filter 1009 may be a filter arranged to preferentially or advantageouslypass light of that colour so as to reject ambient light of othercolours. This can help the optical sensor to isolate changes in thelight received at the light sensor due to movement of the skin fromchanges in environmental lighting conditions at the skin (e.g. due to auser wearing headwear including the optical sensor moving about relativeto a set of environmental light sources). This is particularly relevantwhere the distance from the skin to the optical sensor is severalmillimetres and shrouding is not provided to shield the area of skinbeing sampled by the optical sensor from environmental light. Infra-redlight is particularly advantageous for imaging skin and avoids the skinof the user being visibly illuminated by the optical sensor.

Filters 1008, 1009 may be integrated with the light source and lightsensor, respectively. Additionally or alternatively, filters 1008, 1009may be integrated with the lenses 1001 and 1002, respectively. Thelenses 1001 and 1002 may be provided as a single piece into whichoptional filters 1008 and/or 1009 may be integrated. In the event thatthe light source 1020 is a laser, the laser itself may be polarized—afurther polarizing filter may or may not be required to produce light ofa desired polarization.

One or both of the lenses 1001 and 1002 may be adjustable so as toenable optimal performance to be maintained over a range of distances1005 between the optical flow sensor 1000 and the skin 1004. Forexample, lens 1002 may have a variable focal length so as to accommodatea range of distances 1005 whilst ensuring fine detail illuminated at theskin is captured at the light sensor 1022. Typically, however, it ispossible to achieve good performance over a range of distances 1005(e.g. a range of several millimetres for a distance of around 5 mm) witha fixed focal length lens 1002.

A window 1003 may be provided within the housing in which the opticalsensor is located. In FIG. 10, the housing is shown by way of example asthe frame body 101 of the glasses shown in FIG. 1. The window ispreferably or advantageously transparent to the frequency of lightemitted by the light source 1020. The window may support a coatingdesigned to prevent reflection of the colour of light emitted by thelight source 1020. Preferably or advantageously the window is one andthe same as the lenses 1001, 1002; additionally or alternatively thewindow may include the filters 1008, 1009.

Preferably or advantageously, the light source 1020 is a laser. Thecoherent nature of laser light is particularly suitable for revealingfine detail in the skin and hence allowing accurate tracking of skinmovements. The laser could be a vertical-cavity surface-emitting laser(VCSEL) which offers a compact, low power light source suitable for usein a battery-powered device.

The processor 1024 may further control the illumination of the skin bythe light source 1020. The processor 1024 and the light sensor 1022 maybe provided as a single integrated circuit. The light sensor 1022 may beconfigured to capture an array of pixels, e.g. a 16×16 array, 18×18 or30×30 array of pixels. A resolution at the skin of around 0.1 mm hasbeen found to offer good performance—this can be achieved by providing asuitable number of pixels and a suitable lens 1002 given the sensor-skindistance 1005.

In some examples, more than one optical sensor may be provided.

In a recognition phase, the area of the skin can be imaged by theoptical sensor, and the captured image processed in dependence on thestored data set to determine a correspondence between the captured imageand the skin characteristics for the one or more user. Suitably, thecaptured image, and hence the user, can be authenticated by associatingthe captured image with one of the one or more user, in dependence onthe determined correspondence.

Suitably, the optical sensor does not move relative to the headwear, forexample the frame body, although in some examples it could move relativeto the headwear, for example to image a different area.

In preferred or advantageous examples, the optical sensor is configuredto capture a series of images of an area of skin of a user. This canreveal dynamic features that a single static image may not reveal.

The image and/or series of images can be processed as an image signal,which can include for example a signal representative of the imageand/or series of images.

As a facial expression is performed, muscle activity in the face causesthe skin to move. Suitably therefore the optical sensor is configured toimage an area of skin such that the skin will, during a facialexpression, move past the optical sensor. In this way, the opticalsensor is able to image an area of skin that is greater than the fieldof view of the optical sensor. This is schematically illustrated in FIG.9. The optical sensor can image an area of a particular size within itsfield of view 900. In FIG. 9A, a first image taken at a first timeimages a first area of skin. The shaded areas (which might representpixels or groups of pixels) 902, 904 show a feature of the skin in thefirst image. This feature can be any feature mentioned above, orcombination of such features.

FIG. 9B shows a second image taken at a later, second time. Since theskin has moved relative to the sensor between the first time and thesecond time, the second image is of a second area of skin. The secondarea of skin partially overlaps the first area of skin. The feature atthe shaded areas 902, 904 in the second area is shown to have movedupwards relative to its position in the first image. Similarly, FIG. 9Cshows a third image taken at a later, third time. Since the skin hasmoved relative to the sensor between the second time and the third time,the third image is of a third area of skin. The third area of skinpartially overlaps the second area of skin. The feature at the shadedareas 902, 904 in the third area, is shown to have moved upwardsrelative to its position in the second image.

FIGS. 9B and 9C reveal another shaded area 906 which is not visible inthe first image since it lies outside the first area. The series ofimages shown in FIGS. 9A to 9C therefore permit an area greater than anyof the first, second or third areas on their own to be imaged. Thus anoptical sensor with a particular field of view can be used to image anarea of the skin greater than that field of view.

The presence in the first, second and third images of the same referencepoints (the shaded areas 902, 904) permit the three images to becombined (or ‘stitched’) together. This can form a composite image.Suitably the series of images overlap at least an adjacent image in theseries, or abut an adjacent image in the series, such that the imagesmay be combined together. The images may be combined using anyappropriate image processing technique. This can permit smaller and/orcheaper optical sensors to be used to image larger areas of skin thanwould be possible by taking single images.

Suitably the series of images are captured by the optical sensor at arate that permits the areas of the skin imaged by each of the series ofimages to overlap or abut one another. For example, for typical skinmovement speeds during expressions mentioned herein, a frame capturerate of 30 frames per second (fps), or approximately 30 fps may besuitable. The frame rate can be increased or decreased as appropriate,for example, facial expressions that result in slower facial skinmovement can mean that reduced frame rates are sufficient to capture therequired data that may be necessary.

In some examples, the frame rate at which the series of images iscaptured is at least about 30 frames per second (fps). Suitably theframe rate is greater than about 30 fps. Suitably the frame rate isgreater than about 50 fps, or greater than about 80 fps, for exampleabout 90 fps. Higher frame rates can be used. Higher frame rates arelikely to result in capturing a series of images which more accuratelyrepresent, or track movement of, the areas of the skin, particularly forexpressions involving faster facial skin movement.

Suitably, the taking of a series of images additionally or alternativelypermits dynamic information to be determined, for example how a wrinkledevelops over time. An example of this is a wrinkle at the corner of theeye that develops when a person smiles. The transient development of thewrinkle will be different for different people. The wrinkle may developfaster in some people than others. The wrinkle may develop to become alarger wrinkle in some people than others. The location of the wrinkleis also likely to differ between people. Detecting how the wrinkle (ormore generally, any facial feature or combination of facial features)develops over time can permit user-identifiable information to beobtained which can be used, as discussed herein, to form a data setand/or to authenticate a user against a data set. Another example ofdynamic information would be how far and/or how fast a feature such as apigmented area moves. Another example is how the shape of a pigmentedarea changes over time during a facial expression. Many other examplesare possible.

Suitably the captured image, or image signal, can be processed. Suitablythe captured image, or image signal, can represent a time-varyingsignal. The processing can include any suitable image processingtechnique. Suitably the processing can include one or more processingtechnique as described herein.

FIG. 13 illustrates the skin microstructure of a region 1301 captured byan optical sensor 1303 located at the temple of a pair of glasses 1302.Biometric information describing the microstructure of region 1301 forone or more users may be held by a biometric system. The informationcould be an image of the region 1301 (potentially processed so as toenhance or emphasise features such as wrinkles, blemishes, skin texture,etc.) or any other information characterising features of the region ofskin 1301. As illustrated in the figure, a region of skin for whichbiometric information is to be held for users and which is to becaptured by an optical sensor in order to authenticate a user may besubstantially larger than the area imaged by an optical sensor which, inuse, is arranged at headwear so as to image that region. For example, inFIG. 12, Images A and B illustrate images captured by optical sensorswithin a larger region 1301 which may be used to perform biometricidentification. In a first position labelled (a) in the figure, thesensor captures the area of region 1301 which is shown in Image A. In asecond position labelled (b) in the figure, the sensor captures the areaof region 1301 which is shown in Image B.

A user for which information representing region 1301 is stored at abiometric system may be identified from either of the images A and Bshown in FIG. 13. This enables biometric identification to be performedwhen the optical sensor is at a range of positions relative to theuser's head. Variations in the position of optical sensors relative tothe head of a user can be expected because from one use to the nextheadwear will typically not be replaced in precisely the same position,and during use some movement relative to the face can be expected tooccur (e.g. slippage of a pair of glasses down the nose). In the exampleshown in FIG. 13, optical sensor 1303 takes two positions relative tothe skin in the temple area: at the position shown in (a) the opticalsensor captures Image A, and at the position shown in (b) the opticalsensor captures Image B. In this example the movement of the glassescould be due to slippage of the glasses down the nose and hence themovement of the glasses relative to the skin as indicated by the arrowin the image of region 1301 shown in the figure.

The headwear 100, 200 may include one or more lens adapted to, in usewhen the headwear is worn, focus light reflected from an area of skinonto the optical sensor. The optical sensor may include a diffractiongrating adapted to create a known diffraction pattern on the sensor soas to enable lens-less image capture.

The headwear may include one or more light sources arranged toilluminate the area or areas of skin in the field of view of the opticalsensor. The one or more light sources may be adapted to illuminate thearea(s) of skin with non-visible light. The one or more light sourcesmay be one or more lasers suitable for use at human skin. At least oneof the one or more light sources may be substantially monochromatic andthe optical sensor may include a filter adapted to preferentially oradvantageously pass light of the colour emitted by the light source.

The optical sensor may be configured to receive light reflected from thearea of skin through a plane-polarized filter oriented such that, in usewhen the headwear is worn, the plane of polarization of the filter issubstantially parallel to the plane of the area of skin.

The optical sensor and any optical elements in the light path between anarea of skin and the optical sensor may be arranged so as to not contactthe skin when the apparatus is worn.

The areas of skin imaged by the optical sensor may be no larger than 20square millimetres, preferably or advantageously no larger than 15square millimetres, and most preferably or advantageously no larger than10 square millimetres.

In some examples, the system may include sensors for sensing electricalmuscle activity and optical sensors for sensing skin characteristics.The system may authenticate a user using the electrical muscle activitysensors and/or the optical sensors. Suitably, the system may beconfigured to authenticate the user to a higher level (or to provide fora greater level of security) by requiring that authentication beprovided by both the electrical muscle activity sensors and the opticalsensors.

In some examples, authentication may still be provided by the systemwhere authentication would have been refused by using may be only one ofthe electrical muscle activity sensors or the optical sensors. Forexample, where the system uses the electrical muscle activity sensors(or the optical sensors) to generate an authentication result that failsby a particular amount or tolerance, but the system uses the opticalsensors (or the electrical muscle activity sensors) to generate apositive authentication result, the overall result may be positive. Thisis suitably the case where the failure amount or tolerance is withinpredetermined levels. These predetermined levels can be selected toensure that the levels required that may be necessary remain associatedwith the respective user. For example, the tolerance might suitably bewithin 15%, 10% or 5% of the respective threshold for that user.

Where authentication is based on the behaviour of multiple features(such as signal features derived from channels of EMGs and/or opticalimage features), the system can be arranged to authenticate a userwhere, say, three features from EMG data are satisfied or where, say,two features from EMG data and one optical image feature are satisfied.The numbers of features that may be required, and the numbers offeatures from each source (i.e. EMG or optical image) can be varied asdesired to obtain the required level of security that may be necessary.

A particularly advantageous approach to enrolling or calibrating a userusing the headwear is to treat the outputs of the sensors (preferably oradvantageously information describing the time-varying signals output bythe or each sensor) as inputs to a machine learning system. By causing auser to perform different facial expressions and informing the machinelearning system which facial expressions are being activated, themachine learning system can (using suitable algorithms) learn toidentify what set of inputs (and optionally their variation over time)are associated with each facial expression. The machine learning systemmay provide a dataset for use by the processor so as to enable theprocessor to authenticate the time-varying signal as being associatedwith one or more users, according to the scheme learned by the machinelearning system.

A machine learning system for learning electrical muscle activity and/ordynamic skin information, associated with one or more facialexpressions, may be provided at a computer system to which the headwear100, 200 is connectable, and the headwear may be provided with alearning mode (which might be a calibration mode) in which the outputsof the sensors are passed through to the machine learning system at thecomputer system. A dataset formed by such a machine learning system maybe stored at a data store (suitably at the computer system, such as atthe remote device 400) for use by the processor. Such a dataset couldinclude, for example, an algorithm or set of parameters for a predefinedalgorithm, executable code for the processor, and so on.

The machine learning algorithm or model (or the algorithm trained bymachine learning or the model formed using machine learning) need notuse pattern recognition. In some examples, the machine learningalgorithm can be used to distinguish between users in a population asthe users perform the same expression or series of expressions. Thealgorithm is suitably trained by machine learning to be able todifferentiate between the users, for example by optimising or maximisingthe differences between outputs of the algorithm or model.

Biometric identification of a user may be performed using headwearincluding one or more sensors configured for determining the size of theuser's head and/or one or more of their gross features, such as theposition of their ears. For example, proximity sensors may be providedat headwear at locations so as to capture a measure of the distancebetween the sensor and the skin of the temple region. For headwearapparatus in which such sensors are located at fixed positions, in usewhen the apparatus is in position, such distance information isrepresentative of the size of the user's head.

One or more further proximity sensors may be provided in order to detectwhen the headwear is in the correct position on the user's face. Forexample, in the example of a pair of glasses, a proximity sensor (e.g.around the position of sensor 108 in FIG. 1) may be provided at thebridge of the glasses over the glabellar muscle in order to detect whenthe glasses are located in the correct position (e.g. sitting high onthe bridge of the nose close to the face). The biometric system may beconfigured to measure distance to the temple skin from proximity sensorson the arms of the glasses when the distance from the glasses to theskin overlying the glabellar muscle as measured by the respectiveproximity sensor indicates that the distance is within a predefinedrange—that predefined range may indicate that the glasses are in thecorrect position.

Additionally or alternatively, proximity sensors may be provided atlocations other than over the temple so as to form measurements of thedistance from those respective sensors to the head and hence provide anindication of the head size which may be used as a biometric identifierof the user.

Additionally or alternatively, sensors may be provided to detect theposition of the ears when the headwear is worn in a predeterminedposition by a user. For example, a pair of glasses 1203 are shown inFIG. 12 which include one or more sensors for determining the positionof the ear relative to the glasses. A contact sensor 1202 may beprovided to determine which part of the arm of the glasses rests incontact with the ear. The contact sensor may be any kind of sensor fordetecting contact with the skin. For example, the contact sensor couldbe a set capacitive sensing elements arranged along the arm of theglasses in the region in which it is expected the glasses will rest onthe ear and configured so as to indicate which of the elements are incontact with the skin—since the position of the elements on the arm ofthe glasses is known, the elements which are in contact with the earskin indicate where the ear is relative to the position of the glasses.When the glasses are worn in the correct position, the position of theear may therefore be determined and hence used as a biometric identifierof the user.

The sensors used to detect the position of the ears may alternatively oradditionally include one or more proximity sensors 1201 mounted at thedistal end of the arms of the glasses and arranged so as to measure thedistance from the sensors to post-auricular skin behind the ear. Sincethe position of the proximity sensors on the arms of the glasses isknown, the distance between the proximity sensors and the skin behindthe ear indicates where the ear is relative to the position of theglasses. When the glasses are worn in the correct position, the positionof the ear may thus be determined and used as a biometric identifier ofthe user.

The proximity sensors may be any suitable kind of sensor—for example,one or more of an ultrasound sensor configured to measure distance basedon time of flight and/or phase changes detected between a transmittedand received reflected signal, a laser or other light source configuredto illuminate the skin with divergent beams such that the separation ofa received reflected signal is indicative of the distance the light hastravelled, and a light source and light intensity detector configured todeterminer the distance to the skin from the intensity of the reflectedlight from the light source which is received at the light intensitydetector from the skin.

It may be determined that the glasses are in the correct positionthrough the use of a proximity sensor at the bridge of the glasses inthe manner described above. Such a proximity sensor could be one and thesame as the proximity sensor described above for causing at least someparts of the sensor system to enter a sleep state (e.g. on the distancethat sensor measures to skin overlying the glabellar exceeding somepredefined threshold) or powering up (e.g. on the distance that sensormeasures to skin overlying the glabellar falling below some predefinedthreshold, and optionally on the arms of the glasses being extended asdetected by sensors at hinges at the arms of the glasses).

More generally, the glasses described herein by way of example could beany kind of headwear, including a helmet, virtual or augmented realityheadset, and headphones.

Biometric identifiers formed based on the size of a user's head relativeto headwear including a set of one or more sensors at fixed positions onthe headwear may be used as a crude biometric that roughly identifiesthe user. Such a biometric can be useful where security is not paramountand a quick determination as to which user is wearing the headwear ismay be required—for example, the headwear could be a VR headset and thebiometric identification described herein may be performed in order toselect between different avatars representing each user in a computergame.

In some examples, the system is suitably configured to determine thegaze of a user. In other words, the system is able to determine adirection in which a user is looking. This can be done in one of severalways. The headwear can for example include sensors forelectrooculography (EOG). These sensors can include additional sensors,such as dedicated EOG sensors, or they can include other sensors such asEMG sensors, or electric potential sensors, which can be used to detectthe orientation of the eye within the socket, and hence determine thedirection of gaze of a user. Suitably, the sensors are configured todetect differences between the relatively higher charge towards the backof the eye and the relatively lower charge towards the front of the eye.These differing charges produce a detectable dipole, which can be usedto infer the user's gaze.

Additionally or alternatively, the headwear can include one or moremotion sensor, such as an inertial measurement unit, configured todetermine the movement of the headwear in three dimensions. This caneither assume that a user is looking along a direction which is fixedrelative to the orientation of the headwear (and so determine the user'sgaze from the orientation of the headwear), or it can take into accounta user's direction of gaze as determined from, for example, eye dipolemeasurements to vary this direction (such that the user's gaze isdetermined from the combination of headwear orientation and where theeyes are looking relative to the headwear orientation). This combinedapproach can lead to a more accurate system, and one which is morenatural to a user of the system.

Detection of a user's gaze in this way can be used to be able to chooseor select between options. For example, this approach can be used in anauthentication process (either separately from or together with otherauthentication processes described herein).

In some examples, eye gaze can be used to enter a password or personalidentification number (PIN). This is illustrated in FIG. 11A, whichshows a person wearing headwear 1100 for determining the 3D orientationof the headwear. FIG. 11A shows use of VR headwear. A virtual display(illustrated for ease of understanding separately at 1102) is configuredto display a number of options: here, the display displays the numbers 1to 8. Any other set of numbers, letters, characters, symbols and/orobjects in 3D space and so on could be used. In general, the display candisplay a number of items for selection. The user directs their gazealong the series of items as shown by the gaze path 1104. The user canperform a selection action when the gaze falls upon an item which theuser wishes to select. The selection action can take any appropriateform. For example, the selection action can include the user pausingwhile looking at the desired item for selection, pressing a button,performing a facial expression and so on. Suitably feedback is providedto the user when an item is selected. This feedback can take anysuitable form such as visual feedback, for example the item beinghighlighted, audio feedback, for example a beep, haptic feedback, forexample a vibration, and so on. In this approach, the user is able tolook at the items in any order, and to select may be only the desireditems by performing the selection action.

In some examples, the selection action is a jaw clench (i.e. detectionof the characteristic electrical muscle activity associated with a jawclench). This has the advantage that it is discreet, and so less likelyto be noticed by a third party observer as the user makes the selection.

Combining eye gaze with headwear orientation, and/or selecting an optionusing a discreet expression, can assist in obscuring the selected optionfrom third parties. This can assist in improving the security of thesystem, for example by helping to protect the user's PIN.

In another example, as illustrated in FIG. 11B, a user 1100 may gaze ata single item on the display 1102, and make a selection action when theyare looking at the item to be selected. Following the selection, theuser may gaze at another item. The display may change after one or moreselection actions have been made, to present the user with other itemsor combinations of items for selection. This can help to increase thecomplexity of the password that the user enters, and so improvesecurity.

In another example, illustrated in FIG. 11C, one or more items can moveor scroll across the display 1102, as indicated by the arrows 1106,1108. In this example the user may keep their gaze constant or still,and can perform the selection action when the desired item for selectionpasses across the direction of their gaze. In applications such as VRheadsets, where the display is not visible to third parties, this canhelp obfuscate the selections being made by the user, since the thirdparty is not able to determine the items that the user is looking atwhen making their selection (which may be possible where the user's headmovements are visible). In a modification to this example, asillustrated in FIG. 11D, multiple rows can move (as indicated by thearrows 1110) across the path of a user's gaze so that more than one itemat a time can be seen by the user along the direction of their gaze.This is represented in FIG. 11D by the row including numbers 7-8-5-6being presented in front of the row including numbers 5-6-7-8, and soon. A selection action can be performed when the desired item appears inthe chosen row, known to the user. A third party observer will not knowin which row the user is selecting items. This can also aid security byproviding further obfuscation to the user-selected item or sequence ofitems.

The level of security can be selected as desired, for example by varyingthe number of items available for selection and/or the number ofgestures may be required for authentication.

The apparatus and system of FIGS. 4A, 4B and 10 are shown as including anumber of functional blocks. This is schematic only and is not intendedto define a strict division between different logic elements of suchentities. Each functional block may be provided in any suitable manner.It is to be understood that intermediate values described herein asbeing formed by sensors or processors need not be physically generatedby the sensor/processor at any point and may merely represent logicalvalues which conveniently describe the processing performed by thesensor/processor between its input and output. The use of the term“image capture” need not imply that an image is formed or output by anoptical sensor or processor and may refer generally to capturingcontrast, texture or other information by an optical sensor which doesnot lead to an image being formed.

Generally, any of the functions, methods, techniques or componentsdescribed above can be implemented in software, firmware, hardware(e.g., fixed logic circuitry), or any combination thereof. The terms“module,” “functionality,” “component”, “element”, “unit”, “block” and“logic” may be used herein to generally represent software, firmware,hardware, or any combination thereof. In the case of a softwareimplementation, the module, functionality, component, element, unit,block or logic represents program code that performs the specified taskswhen executed on a processor. The algorithms and methods describedherein could be performed by one or more processors executing code thatcauses the processor(s) to perform the algorithms/methods. Examples of acomputer-readable storage medium include a random-access memory (RAM),read-only memory (ROM), an optical disc, flash memory, hard disk memory,and other memory devices that may use magnetic, optical, and othertechniques to store instructions or other data and that can be accessedby a machine.

The terms computer program code and computer readable instructions asused herein refer to any kind of executable code for processors,including code expressed in a machine language, an interpreted languageor a scripting language. Executable code includes binary code, machinecode, bytecode, and code expressed in a programming language code suchas C, Java or OpenCL. Executable code may be, for example, any kind ofsoftware, firmware, script, module or library which, when suitablyexecuted, processed, interpreted, compiled, executed at a virtualmachine or other software environment, cause a processor of the computersystem at which the executable code is supported to perform the tasksspecified by the code.

A processor, computer, or computer system may be any kind of device,machine or dedicated circuit, or collection or portion thereof, withprocessing capability such that it can execute instructions. A processormay be any kind of general purpose or dedicated processor, such as aCPU, GPU, System-on-chip, state machine, media processor, anapplication-specific integrated circuit (ASIC), a programmable logicarray, a field-programmable gate array (FPGA), or the like. A computeror computer system may include one or more processors.

The applicant hereby discloses in isolation each individual featuredescribed herein and any combination of two or more such features, tothe extent that such features or combinations are capable of beingcarried out based on the present specification as a whole in the lightof the common general knowledge of a person of ordinary skill in theart, irrespective of whether such features or combinations of featuressolve any problems disclosed herein, and without limitation to the scopeof the claims. The applicant indicates that aspects of some embodimentsmay include or can consist of any such individual feature or combinationof features. In view of the foregoing description it will be evident toa person skilled in the art that various modifications may be madewithin the scope of some embodiments.

The invention claimed is:
 1. A biometric authentication system,comprising: headwear including a plurality of biosensors each configuredto sample muscle activity so as to obtain a respective time-varyingsignal, wherein the biosensors are located on the headwear such that, inuse when worn by a user, the biosensors are configured to detect muscleactivity of a predetermined plurality of facial muscles comprisingcorresponding muscles from each side of the user's face; a data storefor storing a data set representing characteristic muscle activity forone or more users; and a processor configured to: process thetime-varying signals from the biosensors to determine a measure ofasymmetry in the muscle activity of the corresponding muscles from eachside of the user's face; process the time-varying signals from thebiosensors in dependence on the stored data set so as to determine acorrespondence between a time-varying signal and characteristic muscleactivity of one of the one or more users, and in dependence on thedetermined correspondence and the determined measure of asymmetry,authenticate the time-varying signals as being associated with the user.2. The biometric authentication system according to claim 1, wherein thepredetermined plurality of facial muscles includes a muscle that isinvoluntarily co-contracted in at least one facial expression.
 3. Thebiometric authentication system according to claim 1, wherein the dataset includes a stored time-varying signal, and the processor isconfigured to process the time-varying signals in dependence on thestored data set by comparing the time-varying signals to the storedtime-varying signal.
 4. A biometric authentication system according toclaim 3, wherein the processor is configured to compare the time-varyingsignals to the stored time-varying signal by determining whether thesignals are within a predetermined tolerance of one another.
 5. Thebiometric authentication system according to claim 1, wherein the dataset includes one or more stored signal features, and the processor isconfigured to process the time-varying signals in dependence on thestored data set by comparing one or more signal features of thetime-varying signals to the one or more stored signal features.
 6. Thebiometric authentication system according to claim 5, wherein theprocessor is configured to determine whether a threshold number ofsignal features of the time-varying signals correspond to the one ormore stored signal features.
 7. A biometric authentication systemaccording to claim 5, wherein the processor is configured to determinethat a signal feature of the time-varying signals corresponds to astored signal feature where the respective signal features are within apredetermined tolerance of one another.
 8. A biometric authenticationsystem according to claim 5, wherein the processor is configured todetermine that a signal feature of the time-varying signals correspondsto a stored signal feature where the signal feature exceeds a thresholdvalue for that signal feature determined in dependence on the storedsignal feature.
 9. A biometric authentication system according to claim1, wherein the characteristic muscle activity represented by the dataset includes a measure of at least one selected from the groupconsisting of: an amplitude of at least one of the time-varying signals;a variation with time of the amplitude of at least one of thetime-varying signals; a measure of asymmetry in the muscle activity ofcorresponding muscles from each side of the user's face; and the timingof activation of at least one facial muscle, relative to a common timebase or to the timing of activation of at least one other facial muscle.10. A biometric authentication system according to claim 1, wherein theprocessor is configured to determine the correspondence between thetime-varying signals and the characteristic muscle activity by at leastone selected from the group consisting of: applying a patternrecognition algorithm; and applying an analysis of variance test.
 11. Abiometric authentication system according to claim 1, wherein theprocessor is configured to process the time-varying signals to obtain aplurality of representative values of the time-varying signals, the dataset including a plurality of stored representative values, therepresentative values and the stored representative values beinggenerated by a predefined algorithm, wherein the processor is configuredto compare the representative values to the stored representativevalues.
 12. A biometric authentication system according to claim 1,wherein one or more of the biosensors are electric potential sensors.13. A biometric authentication system according to claim 1, wherein theheadwear includes sensors for determining the user's gaze.
 14. Abiometric authentication system according to claim 1, wherein theprocessor is configured to process the time-varying signal by at leastone selected from the group consisting of: clipping the signal; signaldenoising; applying a signal baseline correction; using onset detection;using data segmentation; applying a log-transform; and extracting one ormore features from the signal.
 15. A biometric authentication systemaccording to claim 1, wherein the sampled muscle activity is electricalmuscle activity and the stored data set comprises characteristicelectrical muscle activity for the one or more users.
 16. A biometricauthentication system according to claim 1, wherein the biometricauthentication system is configured to output an authentication signalfor controlling access to a resource.
 17. A method for authenticating auser wearing headwear, the headwear including a plurality of biosensorsconfigured to sample muscle activity comprising corresponding musclesfrom each side of the user's face, the method comprising: sampling,using each of the plurality of biosensors, muscle activity of the userto obtain a respective time-varying signal; processing the time-varyingsignals from the biosensors to determine a measure of asymmetry in themuscle activity of the corresponding muscles from each side of theuser's face; processing the time-varying signals from the biosensors independence on a data set representing characteristic muscle activity forone or more users so as to determine a correspondence between atime-varying signal and characteristic muscle activity of one of the oneor more users; and in dependence on the determined correspondence andthe determined measure of asymmetry, authenticating the time-varyingsignals as being associated with the user.